Jump to content
ryan

Module: Import Pages from CSV file

Recommended Posts

I could not use the date format of d/m/Y that I wanted no mater how hard I tried. As an example of the strange results I was getting here is the code I used on the event page - the results are in the comments :

echo $config->dateFormat; // Y-m-d H:i:s
echo '<br />';
echo $page->event_start;  // 21/01/2012
echo '<br />';
echo $page->event_end;  // 15/04/2012

$date1 = strtotime($page->event_start);
$date2 = strtotime($page->event_end);

echo $date1;  // empty
echo '<br />';
echo $date2; // empty
echo '<br />';

$diff = $date2 - $date1;

$no_days = floor($diff / (60*60*24)) ;

echo 'duration : ' .$no_days;  // 0

This may have been a result of changing the date field formats too many times.

I deleted all the fields and created them again with default US settings. I then changed all my dates in the CSV to US style m/d/Y re-imported and now it works as expected!

I did not try the d-m-Y format yet (I could not find any country that uses this format in the OSX dates preferences) - it was easier to change to US date settings.

On a side note, I always thought the US date format was odd - the logical way would be small > medium > large (day > month > year). Are they are the only country that starts with a month ?

http://en.wikipedia....rmat_by_country

Thanks,

Michael

Share this post


Link to post
Share on other sites

Just discovered there is a simple way to do the d-m-Y dates (or any format) in Apple Numbers without switching your main format settings - you just need to create a custom date format - see attached screenshot :

http://db.tt/SnvwLoA7

Share this post


Link to post
Share on other sites

I don't know why the US date format is the way it is. The d/m/Y format does seem more logical. Maybe the US date format is meant as some kind of calendar hierarchy? The date format I prefer the most is Y-m-d, but mainly because it's string-sortable. :)

You don't necessarily need to go changing settings in your spreadsheet if you don't want to. Another way to convert your d/m/Y dates to d-m-Y (dashes rather than slashes) is to load the CSV into a quality text editor and then use a regexp to swap the slashes for dashes. Here's the search and replace I'd use in TextWrangler (BBEdit):

(\d{2})/(\d{2})/(\d{4})	 =>	\1-\2-\3

PHP's strtotime() function treats dates with dashes as European format and dates with slashes as US format. So once you convert those dates in your CSV to use dashes rather than slashes, they should import just fine with the CSV import module.

Share this post


Link to post
Share on other sites

Thanks for the regexp tip. I am trying to keep a simple workflow for business users where all data preparation can be done in spreadsheets (excel / numbers / google docs) and then imported to Processwire. I'm a big fan of using spreadsheets as a middle ground or starting point for building applications especially where lots of data is involved.

I've not tested yet, but will the import work with the new multilingual fields? In the past, spreadsheets have been a useful way to translate content with multiple people.

Share this post


Link to post
Share on other sites

It's been a while - sorry for that!

I have just tested this module again in a live setup (I was previously testing on a localhost setup). I am still getting this 'problem' where upon installing the module 'Import Pages from CSV' I am simply returned to the module information screen with a highlighted message saying: 'Click to your 'Setup' page to start using the CSV Importer'. I am given no clue as to having to change the 'Process' dropdown for that module setup page in the page tree.

I take it this isn't the designed behaviour?

My localhost testing environment is (if any of this helps) a MAMP installation using PHP 5.3.2 and my live site setup is an Apache installation using using PHP 5.2.17

Share this post


Link to post
Share on other sites

It sounds like we're using the exact same localhost testing environment.

This module creates a page: Admin > Setup > Import pages from CSV

Looking at the install function, it won't assign the process if there is already a page there named 'import-pages-csv'. If you've already got a page there with the same name, you'll have to edit it and select 'ImportPagesCSV' for the 'process' field and then hit save. However, assuming you don't have a page there named 'import-pages-csv', then it should take care of it all for you and be ready to use right after you've installed.

Share this post


Link to post
Share on other sites

Thanks Ryan,

I did copy over a lot of the content from my localhost - so that could explain a page already being there (assuming that's the case). Does uninstalling the module remove it's setup configuration (removing the page and processes)?

It's not a big enough issue to warrant spending much time on it - especially as I now know how to get around it.

Share this post


Link to post
Share on other sites

Uninstalling should remove the page and process. Also there's no harm in manually removing the page either. But if you just want to add the CSV import process, then I would just edit the existing page that's already there and select 'ImportPagesCSV' and save, and that'll be the same as if it had installed it.

Share this post


Link to post
Share on other sites

Thanks a lot for making this module, ryan. I've already used to to import over 400 pages... huge time saver.

The one suggestion I have was touched on by one or two people earlier in the thread: the importer doesn't work when the page is missing a 'title' field, presumably because it's used to generate the page name. It's pretty easy to get around by making a dummy title field, populating it with a CSV column, and then deleting it after importation, but it would be really great if during Step 2 we could specifiy a column from the CSV to use for generating a page name/slug.

Share this post


Link to post
Share on other sites

Idea of ​​such exports is great! And thanks to Ryan for this!

As for me: I'm trying and trying! What is the problem, doc?

What I'm doing wrong or is this a known problem?

If i try to import returns: «Unable to import page because it has no required 'title' field or it is blank.»

My file structure is as follows:

"title"

"Yakimanka"

"Yaroslavl"

"Yasenevo"

What could it be?

Share this post


Link to post
Share on other sites

The module works fine! Thanks again Ryan!

For example, the contents of my .csv file, I translated from Rus to Eng.

Obtained the «title» can not be other than English at import?

Share this post


Link to post
Share on other sites

Thanks AlexV! This module was written before multi-language support in PW, and hasn't yet been updated to support multi language title/text/textarea fields (though does support multi language alternate fields). However, even if it had been updated, you would still need a title specified for the page in the 'default' language (since it is a required field). It doesn't care what language the 'default' language is, whether English, Russian, etc. But when you are importing with this module, it is assuming that default language for any multi-language fields. It sounds like in your case, you may want to have Russian as your default language, and English as an add-on language. That way you don't have to translate to English before importing.

Share this post


Link to post
Share on other sites

Thank you Ryan for your reply! I did not use multi-language support before, but your words open new opportunities for me.

Share this post


Link to post
Share on other sites

Most likely you got a page title that is empty, thus the error. Every page in PW needs at least the "title" field which is mandatory. You can't save a page without giving it a title. Is there any entries with no title in your csv?

Share this post


Link to post
Share on other sites

Thanks Soma, I received a reply. I used in the 'title' rus characters and did not use multi-language support, so when exporting 'title' field was obtained «as a blank».

Share this post


Link to post
Share on other sites

Ryan,

Many thanks for this module, I just used it to add 436 users.

There were a small handful of things that weren't initially clear, so I detailed them below for anyone else trying to import users.

If you plan to import passwords, you need to open the module and add FieldTypePassword to $fieldtypes

protected $fieldtypes = array(
'FieldtypePageTitle',
'FieldtypeText',
'FieldtypeTextarea',
'FieldtypeInteger',
'FieldtypeFloat',
'FieldtypeEmail',
'FieldtypeURL',
'FieldtypeCheckbox',
'FieldtypeFile',
'FieldtypePassword', // add this line
);

Since users are pages and all pages require a title, your CSV will need to have a title column.

In my case, I duplicated all the usernames into that column — so name and title are the same.

In order for title to show as a connection option during your import, you need to add the title field to the user template file.

To do this, go to: Setup > Templates (open the filters area at the top, and choose "show system templates". Select the user template and add the title field.

One other thing to note, be sure to have a roles column in your CSV with roles for each user.

I forgot that during my first test import and all the users were set to guest.

You should be all set to import your users.

:)

  • Like 8

Share this post


Link to post
Share on other sites

Quick question -- is it possible to automatically import a CSV file from a specified URL, using this module's functions and Lazy Cron? I couldn't figure out how on my cursory glance over the module's code. I'm trying to see if this would be suitable for updating an online inventory, or if I'd be better off rolling my own solution. Thanks!

Share this post


Link to post
Share on other sites

The module isn't intended for non-interactive use. However, the PW API certainly is (and LazyCron), and pretty clean and simple for doing stuff like this.

Share this post


Link to post
Share on other sites

Is there any way to insert file / image descriptions while importing? If not, any idea how one could implement a feature like that? :)

Share this post


Link to post
Share on other sites

For something like that, I think you'll be better off just importing from the API. Probably 70% of the time that I need to do any kind of importing of anything, I just use the API because it usually makes a quick and easy job of it, without having to consider any limitations. But if you want to get image/file descriptions in with the CSV import module, then I would probably create a temporary field to store them in, for import purposes. Then go back and paste a little code in a template to quickly convert them over:

foreach($mypages as $p) {
   $p->of(false); 
   $p->image->description = $p->temp; 
   $p->save();
}

Once you've done that, you could then go and delete that 'temp' field from the template.

  • Like 3

Share this post


Link to post
Share on other sites

Question: does this import module work with the repeater module? How might the CSV look like if I wanted to import multiple fields which were part of a repeater?

Thanks in advance!

Share this post


Link to post
Share on other sites

Welcome to the forums Barry. I don't think the CSV importer could be used with the repeater field type. Too many components to put together in making that one work through a simple CSV importer. I think you'd be better off using the API on that one. See here for repeater API usage (bottom of page).

Share this post


Link to post
Share on other sites

Is it possible yet to import a set of Pages to represent categories, then import a second set of entries including page references to those categories? I'm looking to import a large EE site with a lot of related entries. I imagine I'd import the category pages with their legacy identifier in a simple text field, and then have the importer match another field in the second import with that of the first (or something similar).

Thanks!

Share this post


Link to post
Share on other sites

@bcartier: The ImportPagesCSV-module can't do this as is. But I tried making a tiny addition to make it support FieldtypePage (those used to make page references) and it worked amazingly well. The only change needed was to add 'FieldtypePage' to $fieldtypes array (just before init() function if you take a look at the module file), like this:


       protected $fieldtypes = array(
               'FieldtypePageTitle',
               'FieldtypeText',
               'FieldtypeTextarea',
               'FieldtypeInteger',
               'FieldtypeFloat',
               'FieldtypeEmail',
               'FieldtypeURL',
               'FieldtypeCheckbox',
               'FieldtypeFile',
               'FieldtypePage', // add this line
               );

After that addition it's possible to choose a Page field when connecting the fields from the CSV to the ones in the chosen template. I had pre-populated categories at the target site and used their id's in the CSV file to reference those categories. Multiple categories worked like a charm as well, just use a pipe in between id's (123|456|789).

Moreover, if you've got only one category per entry to reference, then you don't even need the id's of the categories - you can use paths as well. Here's a little example:

cat.csv:

title
one
two
three
four

entries.csv:

title,categories
a,/cats/four/
b,/cats/three/
c,/cats/one/
d,/cats/two/

Import cat.csv using a template for categories with (at least) title field, under a page at /cats/. Then import entries.csv using a template for entries, having a title field and a page field. This should leave you with entries that are connected to categories. I hope this gets you somewhere. :)

@ryan: Looks like page references could be supported very easily. I just used this successfully to import ~3500 pages with category references from an old site to a new PW one. But maybe there's still something else to be done before they're fully supported?

  • Like 14

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By d'Hinnisdaël
      Happy new year, everybody 🥬
      I've been sitting on this Dashboard module I made for a client and finally came around to cleaning it up and releasing it to the wider public. This is how it looks.
      ProcessWire Dashboard

      If anyone is interested in trying this out, please go ahead! I'd love to get some feedback on it. If this proves useful and survives some real-world testing, I'll add this to the module directory.
      Download
      You can find the latest release on Github.
      Documentation
      Check out the documentation to get started. This is where you'll find information about included panel types and configuration options.
      Custom Panels
      My goal was to make it really simple to create custom panels. The easiest way to do that is to use the panel type template and have it render a file in your templates folder. This might be enough for 80% of all use cases. For anything more complex (FormBuilder submissions? Comments? Live chat?), you can add new panel types by creating modules that extend the DashboardPanel base class. Check out the documentation on custom panels or take a look at the HelloWorld panel to get started. I'm happy to merge any user-created modules into the main repo if they might be useful to more than a few people.
       Disclaimer
      This is a pre-release version. Please treat it as such — don't install it on production sites. Just making sure 🍇
      Roadmap
      These are the things I'm looking to implement myself at some point. The wishlist is a lot longer, but those are the 80/20 items that I probably won't regret spending time on.
      Improve documentation & add examples ⚙️ Panel types Google Analytics ⚙️ Add new page  🔥 Drafts 🔥 At a glance / Page counter 404s  Layout options Render multiple tabs per panel panel groups with heading and spacing between ✅ panel wrappers as grid item (e.g. stacked notices) ✅ Admin themes support AdminThemeReno and AdminThemeDefault ✅ Shortcuts panel add a table layout with icon, title & summary ✅ Chart panel add default styles for common chart types ✅ load chart data from JS file (currently passed as PHP array) Collection panel support image columns ✅ add buttons: view all & add new ✅
    • By Pip
      Hi everyone!
      I'm trying out the Login/Register module for my site. Noted that the module assigns the newly registered user to login-register role. 
      Once you modify the login-register role's permissions, particularly adding page-edit, the new member role will be set to guest. 
      Thing is I'd like to grant my new users the power to create their own pages. Any advice? 
      Thanks. 
    • By Gadgetto
      SnipWire - Snipcart integration for ProcessWire
      Snipcart is a powerful 3rd party, developer-first HTML/JavaScript shopping cart platform. SnipWire is the missing link between Snipcart and the content management framework ProcessWire.
      With SnipWire, you can quickly turn any ProcessWire site into a Snipcart online shop. The SnipWire plugin helps you to get your store up and running in no time. Detailed knowledge of the Snipcart system is not required.
      SnipWire is free and open source licensed under Mozilla Public License 2.0! A lot of work and effort has gone into development. It would be nice if you could donate an amount to support further development:

      Status update links (inside this thread) for SnipWire development
      2020-07-03 -- SnipWire 0.8.7 (beta) released! Fixes some small bugs and adds an indicator for TEST mode 2020-04-06 -- SnipWire 0.8.6 (beta) released! Adds support for Snipcart subscriptions and also fixes some problems 2020-03-21 -- SnipWire 0.8.5 (beta) released! Improves SnipWires webhooks interface and provides some other fixes and additions 2020-03-03 -- SnipWire 0.8.4 (beta) released! Improves compatibility for Windows based Systems. 2020-03-01 -- SnipWire 0.8.3 (beta) released! The installation and uninstallation process has been heavily revised. 2020-02-08 -- SnipWire 0.8.2 (beta) released! Added a feature to change the cart and catalogue currency by GET, POST or SESSION param 2020-02-03 -- SnipWire 0.8.1 (beta) released! All custom classes moved into their own namespaces. 2020-02-01 -- SnipWire is now available via ProcessWire's module directory! 2020-01-30 -- SnipWire 0.8.0 (beta) first public release! (module just submitted to the PW modules directory) 2020-01-28 -- added Custom Order Fields feature (first SnipWire release version is near!) 2020-01-21 -- Snipcart v3 - when will the new cart system be implemented? 2020-01-19 -- integrated taxes provider finished (+ very flexible shipping taxes handling) 2020-01-14 -- new date range picker, discount editor, order notifiactions, order statuses, and more ... 2019-11-15 -- orders filter, order details, download + resend invoices, refunds 2019-10-18 -- list filters, REST API improvements, new docs platform, and more ... 2019-08-08 -- dashboard interface, currency selector, managing Orders, Customers and Products, Added a WireTabs, refinded caching behavior 2019-06-15 -- taxes provider, shop templates update, multiCURL implementation, and more ... 2019-06-02 -- FieldtypeSnipWireTaxSelector 2019-05-25 -- SnipWire will be free and open source Plugin Key Features
      Fast and simple store setup Full integration of the Snipcart dashboard into the ProcessWire backend (no need to leave the ProcessWire admin area) Browse and manage orders, customers, discounts, abandoned carts, and more Multi currency support Custom order and cart fields Process refunds and send customer notifications from within the ProcessWire backend Process Abandoned Carts + sending messages to customers from within the ProcessWire backend Complete Snipcart webhooks integration (all events are hookable via ProcessWire hooks) Integrated taxes provider (which is more flexible then Snipcart own provider) Useful Links
      SnipWire in PW modules directory SnipWire Docs (please note that the documentation is a work in progress) SnipWire @GitHub (feature requests and suggestions for improvement are welcome - I also accept pull requests) Snipcart Website  

       
      ---- INITIAL POST FROM 2019-05-25 ----
       
    • By Sten
      Hello
      Till now I hacked something with the twig template but it works no more with new PW versions so I look forward to create a module. I am working on a site in multiple languages : French, English, Italian, German, Spanish, Portuguese, Hebrew, Russian. The new posts are entered in any language with a field for language. Till now, I got twig files to get the translations with constants defined for each part of the pages.
      So I'd like to create a module to include theses files added according to the url /fr/en/...
      Have you some observations to do before I begin about the direction to take ?
      Thank you
    • By ukyo
      Mystique Module for ProcessWire CMS/CMF
      Github repo : https://github.com/trk/Mystique
      Mystique module allow you to create dynamic fields and store dynamic fields data on database by using a config file.
      Requirements
      ProcessWire 3.0 or newer PHP 7.0 or newer FieldtypeMystique InputfieldMystique Installation
      Install the module from the modules directory:
      Via Composer:
      composer require trk/mystique Via git clone:
      cd your-processwire-project-folder/ cd site/modules/ git clone https://github.com/trk/Mystique.git Module in live reaction with your Mystique config file
      This mean if you remove a field from your config file, field will be removed from edit screen. As you see on youtube video.
      Using Mystique with your module or use different configs path, autoload need to be true for modules
      Default configs path is site/templates/configs/, and your config file name need to start with Mystique. and need to end with .php extension.
      Adding custom path not supporting anymore !
      // Add your custom path inside your module class`init` function, didn't tested outside public function init() { $path = __DIR__ . DIRECTORY_SEPARATOR . 'configs' . DIRECTORY_SEPARATOR; Mystique::add($path); } Mystique module will search site/modules/**/configs/Mystique.*.php and site/templates/Mystique.*.php paths for Mystique config files.
      All config files need to return a PHP ARRAY like examples.
      Usage almost same with ProcessWire Inputfield Api, only difference is set and showIf usage like on example.
      <?php namespace ProcessWire; /** * Resource : testing-mystique */ return [ 'title' => __('Testing Mystique'), 'fields' => [ 'text_field' => [ 'label' => __('You can use short named types'), 'description' => __('In file showIf working like example'), 'notes' => __('Also you can use $input->set() method'), 'type' => 'text', 'showIf' => [ 'another_text' => "=''" ], 'set' => [ 'showCount' => InputfieldText::showCountChars, 'maxlength' => 255 ], 'attr' => [ 'attr-foo' => 'bar', 'attr-bar' => 'foo' ] ], 'another_text' => [ 'label' => __('Another text field (default type is text)') ] ] ]; Example:
      site/templates/configs/Mystique.seo-fields.php <?php namespace ProcessWire; /** * Resource : seo-fields */ return [ 'title' => __('Seo fields'), 'fields' => [ 'window_title' => [ 'label' => __('Window title'), 'type' => Mystique::TEXT, // or InputfieldText 'useLanguages' => true, 'attr' => [ 'placeholder' => __('Enter a window title') ] ], 'navigation_title' => [ 'label' => __('Navigation title'), 'type' => Mystique::TEXT, // or InputfieldText 'useLanguages' => true, 'showIf' => [ 'window_title' => "!=''" ], 'attr' => [ 'placeholder' => __('Enter a navigation title') ] ], 'description' => [ 'label' => __('Description for search engines'), 'type' => Mystique::TEXTAREA, 'useLanguages' => true ], 'page_tpye' => [ 'label' => __('Type'), 'type' => Mystique::SELECT, 'options' => [ 'basic' => __('Basic page'), 'gallery' => __('Gallery'), 'blog' => __('Blog') ] ], 'show_on_nav' => [ 'label' => __('Display this page on navigation'), 'type' => Mystique::CHECKBOX ] ] ]; Searching data on Mystique field is limited. Because, Mystique saving data to database in json format. When you make search for Mystique field, operator not important. Operator will be changed with %= operator.
      Search example
      $navigationPages = pages()->find('my_mystique_field.show_on_nav=1'); $navigationPages = pages()->find('my_mystique_field.page_tpye=gallery');
×
×
  • Create New...