Jump to content
ryan

Module: Import Pages from CSV file

Recommended Posts

I could not use the date format of d/m/Y that I wanted no mater how hard I tried. As an example of the strange results I was getting here is the code I used on the event page - the results are in the comments :

echo $config->dateFormat; // Y-m-d H:i:s
echo '<br />';
echo $page->event_start;  // 21/01/2012
echo '<br />';
echo $page->event_end;  // 15/04/2012

$date1 = strtotime($page->event_start);
$date2 = strtotime($page->event_end);

echo $date1;  // empty
echo '<br />';
echo $date2; // empty
echo '<br />';

$diff = $date2 - $date1;

$no_days = floor($diff / (60*60*24)) ;

echo 'duration : ' .$no_days;  // 0

This may have been a result of changing the date field formats too many times.

I deleted all the fields and created them again with default US settings. I then changed all my dates in the CSV to US style m/d/Y re-imported and now it works as expected!

I did not try the d-m-Y format yet (I could not find any country that uses this format in the OSX dates preferences) - it was easier to change to US date settings.

On a side note, I always thought the US date format was odd - the logical way would be small > medium > large (day > month > year). Are they are the only country that starts with a month ?

http://en.wikipedia....rmat_by_country

Thanks,

Michael

Share this post


Link to post
Share on other sites

I don't know why the US date format is the way it is. The d/m/Y format does seem more logical. Maybe the US date format is meant as some kind of calendar hierarchy? The date format I prefer the most is Y-m-d, but mainly because it's string-sortable. :)

You don't necessarily need to go changing settings in your spreadsheet if you don't want to. Another way to convert your d/m/Y dates to d-m-Y (dashes rather than slashes) is to load the CSV into a quality text editor and then use a regexp to swap the slashes for dashes. Here's the search and replace I'd use in TextWrangler (BBEdit):

(\d{2})/(\d{2})/(\d{4})	 =>	\1-\2-\3

PHP's strtotime() function treats dates with dashes as European format and dates with slashes as US format. So once you convert those dates in your CSV to use dashes rather than slashes, they should import just fine with the CSV import module.

Share this post


Link to post
Share on other sites

Thanks for the regexp tip. I am trying to keep a simple workflow for business users where all data preparation can be done in spreadsheets (excel / numbers / google docs) and then imported to Processwire. I'm a big fan of using spreadsheets as a middle ground or starting point for building applications especially where lots of data is involved.

I've not tested yet, but will the import work with the new multilingual fields? In the past, spreadsheets have been a useful way to translate content with multiple people.

Share this post


Link to post
Share on other sites

It's been a while - sorry for that!

I have just tested this module again in a live setup (I was previously testing on a localhost setup). I am still getting this 'problem' where upon installing the module 'Import Pages from CSV' I am simply returned to the module information screen with a highlighted message saying: 'Click to your 'Setup' page to start using the CSV Importer'. I am given no clue as to having to change the 'Process' dropdown for that module setup page in the page tree.

I take it this isn't the designed behaviour?

My localhost testing environment is (if any of this helps) a MAMP installation using PHP 5.3.2 and my live site setup is an Apache installation using using PHP 5.2.17

Share this post


Link to post
Share on other sites

It sounds like we're using the exact same localhost testing environment.

This module creates a page: Admin > Setup > Import pages from CSV

Looking at the install function, it won't assign the process if there is already a page there named 'import-pages-csv'. If you've already got a page there with the same name, you'll have to edit it and select 'ImportPagesCSV' for the 'process' field and then hit save. However, assuming you don't have a page there named 'import-pages-csv', then it should take care of it all for you and be ready to use right after you've installed.

Share this post


Link to post
Share on other sites

Thanks Ryan,

I did copy over a lot of the content from my localhost - so that could explain a page already being there (assuming that's the case). Does uninstalling the module remove it's setup configuration (removing the page and processes)?

It's not a big enough issue to warrant spending much time on it - especially as I now know how to get around it.

Share this post


Link to post
Share on other sites

Uninstalling should remove the page and process. Also there's no harm in manually removing the page either. But if you just want to add the CSV import process, then I would just edit the existing page that's already there and select 'ImportPagesCSV' and save, and that'll be the same as if it had installed it.

Share this post


Link to post
Share on other sites

Thanks a lot for making this module, ryan. I've already used to to import over 400 pages... huge time saver.

The one suggestion I have was touched on by one or two people earlier in the thread: the importer doesn't work when the page is missing a 'title' field, presumably because it's used to generate the page name. It's pretty easy to get around by making a dummy title field, populating it with a CSV column, and then deleting it after importation, but it would be really great if during Step 2 we could specifiy a column from the CSV to use for generating a page name/slug.

Share this post


Link to post
Share on other sites

Idea of ​​such exports is great! And thanks to Ryan for this!

As for me: I'm trying and trying! What is the problem, doc?

What I'm doing wrong or is this a known problem?

If i try to import returns: «Unable to import page because it has no required 'title' field or it is blank.»

My file structure is as follows:

"title"

"Yakimanka"

"Yaroslavl"

"Yasenevo"

What could it be?

Share this post


Link to post
Share on other sites

The module works fine! Thanks again Ryan!

For example, the contents of my .csv file, I translated from Rus to Eng.

Obtained the «title» can not be other than English at import?

Share this post


Link to post
Share on other sites

Thanks AlexV! This module was written before multi-language support in PW, and hasn't yet been updated to support multi language title/text/textarea fields (though does support multi language alternate fields). However, even if it had been updated, you would still need a title specified for the page in the 'default' language (since it is a required field). It doesn't care what language the 'default' language is, whether English, Russian, etc. But when you are importing with this module, it is assuming that default language for any multi-language fields. It sounds like in your case, you may want to have Russian as your default language, and English as an add-on language. That way you don't have to translate to English before importing.

Share this post


Link to post
Share on other sites

Thank you Ryan for your reply! I did not use multi-language support before, but your words open new opportunities for me.

Share this post


Link to post
Share on other sites

Most likely you got a page title that is empty, thus the error. Every page in PW needs at least the "title" field which is mandatory. You can't save a page without giving it a title. Is there any entries with no title in your csv?

Share this post


Link to post
Share on other sites

Thanks Soma, I received a reply. I used in the 'title' rus characters and did not use multi-language support, so when exporting 'title' field was obtained «as a blank».

Share this post


Link to post
Share on other sites

Ryan,

Many thanks for this module, I just used it to add 436 users.

There were a small handful of things that weren't initially clear, so I detailed them below for anyone else trying to import users.

If you plan to import passwords, you need to open the module and add FieldTypePassword to $fieldtypes

protected $fieldtypes = array(
'FieldtypePageTitle',
'FieldtypeText',
'FieldtypeTextarea',
'FieldtypeInteger',
'FieldtypeFloat',
'FieldtypeEmail',
'FieldtypeURL',
'FieldtypeCheckbox',
'FieldtypeFile',
'FieldtypePassword', // add this line
);

Since users are pages and all pages require a title, your CSV will need to have a title column.

In my case, I duplicated all the usernames into that column — so name and title are the same.

In order for title to show as a connection option during your import, you need to add the title field to the user template file.

To do this, go to: Setup > Templates (open the filters area at the top, and choose "show system templates". Select the user template and add the title field.

One other thing to note, be sure to have a roles column in your CSV with roles for each user.

I forgot that during my first test import and all the users were set to guest.

You should be all set to import your users.

:)

  • Like 8

Share this post


Link to post
Share on other sites

Quick question -- is it possible to automatically import a CSV file from a specified URL, using this module's functions and Lazy Cron? I couldn't figure out how on my cursory glance over the module's code. I'm trying to see if this would be suitable for updating an online inventory, or if I'd be better off rolling my own solution. Thanks!

Share this post


Link to post
Share on other sites

The module isn't intended for non-interactive use. However, the PW API certainly is (and LazyCron), and pretty clean and simple for doing stuff like this.

Share this post


Link to post
Share on other sites

Is there any way to insert file / image descriptions while importing? If not, any idea how one could implement a feature like that? :)

Share this post


Link to post
Share on other sites

For something like that, I think you'll be better off just importing from the API. Probably 70% of the time that I need to do any kind of importing of anything, I just use the API because it usually makes a quick and easy job of it, without having to consider any limitations. But if you want to get image/file descriptions in with the CSV import module, then I would probably create a temporary field to store them in, for import purposes. Then go back and paste a little code in a template to quickly convert them over:

foreach($mypages as $p) {
   $p->of(false); 
   $p->image->description = $p->temp; 
   $p->save();
}

Once you've done that, you could then go and delete that 'temp' field from the template.

  • Like 3

Share this post


Link to post
Share on other sites

Question: does this import module work with the repeater module? How might the CSV look like if I wanted to import multiple fields which were part of a repeater?

Thanks in advance!

Share this post


Link to post
Share on other sites

Welcome to the forums Barry. I don't think the CSV importer could be used with the repeater field type. Too many components to put together in making that one work through a simple CSV importer. I think you'd be better off using the API on that one. See here for repeater API usage (bottom of page).

Share this post


Link to post
Share on other sites

Is it possible yet to import a set of Pages to represent categories, then import a second set of entries including page references to those categories? I'm looking to import a large EE site with a lot of related entries. I imagine I'd import the category pages with their legacy identifier in a simple text field, and then have the importer match another field in the second import with that of the first (or something similar).

Thanks!

Share this post


Link to post
Share on other sites

@bcartier: The ImportPagesCSV-module can't do this as is. But I tried making a tiny addition to make it support FieldtypePage (those used to make page references) and it worked amazingly well. The only change needed was to add 'FieldtypePage' to $fieldtypes array (just before init() function if you take a look at the module file), like this:


       protected $fieldtypes = array(
               'FieldtypePageTitle',
               'FieldtypeText',
               'FieldtypeTextarea',
               'FieldtypeInteger',
               'FieldtypeFloat',
               'FieldtypeEmail',
               'FieldtypeURL',
               'FieldtypeCheckbox',
               'FieldtypeFile',
               'FieldtypePage', // add this line
               );

After that addition it's possible to choose a Page field when connecting the fields from the CSV to the ones in the chosen template. I had pre-populated categories at the target site and used their id's in the CSV file to reference those categories. Multiple categories worked like a charm as well, just use a pipe in between id's (123|456|789).

Moreover, if you've got only one category per entry to reference, then you don't even need the id's of the categories - you can use paths as well. Here's a little example:

cat.csv:

title
one
two
three
four

entries.csv:

title,categories
a,/cats/four/
b,/cats/three/
c,/cats/one/
d,/cats/two/

Import cat.csv using a template for categories with (at least) title field, under a page at /cats/. Then import entries.csv using a template for entries, having a title field and a page field. This should leave you with entries that are connected to categories. I hope this gets you somewhere. :)

@ryan: Looks like page references could be supported very easily. I just used this successfully to import ~3500 pages with category references from an old site to a new PW one. But maybe there's still something else to be done before they're fully supported?

  • Like 14

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Robin S
      After forgetting the class name of the wonderful AdminPageFieldEditLinks module for what feels like the 100th time I decided I needed to give my failing memory a helping hand...
      Autocomplete Module Class Name
      Provides class name autocomplete suggestions for the "Add Module From Directory" field at Modules > New.
      Requires ProcessWire >= v3.0.16.
      Screencast

      Installation
      Install the Autocomplete Module Class Name module.
      Configuration
      Choose the type of autocomplete options list: "Module class names from directory" or "Custom list of module class names". The latter could be useful if you regularly install some modules and would prefer a shorter list of autocomplete suggestions. The list of class names in the modules directory is generated when the Autocomplete Module Class Name module is installed. It doesn't update automatically (because the retrieval of the class names is quite slow), but you can use the button underneath when you want to retrieve an updated list of class names from the directory. The "fuzzy search" option uses custom filter and item functions for Awesomplete so that the characters you type just have to exist in the module class name and occur after preceding matches but do not need to be contiguous. Uncheck this option if you prefer the standard Awesomplete matching. Custom settings for Awesomplete can be entered in the "Awesomplete options" field if needed. See the Awesomplete documentation for more information.  
      https://github.com/Toutouwai/AutocompleteModuleClassName
      https://modules.processwire.com/modules/autocomplete-module-class-name/
    • By teppo
      MarkupMenu is a markup module for generating menu trees. When provided a root page as a starting point, it generates a navigation tree (by default as a HTML "<ul>" element wrapped by a "<nav>" element) from that point onwards. If you've also provided it with current (active) page, the menu will be rendered accordingly, with current item highlighted and items rendered up to that item and its children (unless you disable the "collapsed" option, in which case the full page tree will be rendered instead).
      Modules directory: https://modules.processwire.com/modules/markup-menu/ GitHub repository: https://github.com/teppokoivula/MarkupMenu Usage
      As a markup module, MarkupMenu is intended for front-end use, but you can of course use it in a module as well. Typically you'll only need the render() method, which takes an array of options as its only argument:
      echo $modules->get('MarkupMenu')->render([ 'root_page' => $pages->get(1), 'current_page' => $page, ]); Note: if you omit root_page, site root page is used by default. If you omit current_page, the menu will be rendered, but current (active) page won't be highlighted etc.
      A slightly more complex example, based on what I'm using on one of my own sites to render a (single-level) top menu:
      echo $modules->get('MarkupMenu')->render([ 'current_page' => $page, 'templates' => [ 'nav' => '<nav class="{classes} menu--{menu_class_modifier}" aria-label="{aria_label}">%s</nav>', 'item_current' => '<a class="menu__item menu__item--current" href="{item.url}" tabindex="0" aria-label="Current page: {item.title}">{item.title}</a>', ], 'placeholders' => [ 'menu_class_modifier' => 'top', 'aria_label' => 'Main navigation', ], 'include' => [ 'root_page' => true, ], 'exclude' => [ 'level_greater_than' => 1, ], ]); Note: some things you see above may not be entirely sensible, such as the use of {menu_class_modifier} and {aria_label} placeholders. On the actual site the "nav" template is defined in site config, so I can define just these parts on a case-by-case basis while actual nav markup is maintained in one place.
      Please check out the README file for available render options. I'd very much prefer not to keep this list up to date in multiple places. Basically there are settings for defining "templates" for different parts of the menu (list, item, etc.), include array for defining rules for including in the menu and exclude array for the opposite effect, classes and placeholders arrays for overriding default classes and injecting custom placeholders, etc. 🙂
      MarkupMenu vs. MarkupSimpleNavigation
      TL;DR: this is another take on the same concept. There are many similarities, but also some differences – especially when it comes to the supported options and syntax. If you're currently using MarkupSimpleNavigation then there's probably no reason to switch over.
      I'd be surprised if anyone didn't draw lines between this module and Soma's awesome MarkupSimpleNavigation. Simply put, I've been using MSN (...) for a number of years, and it's been great – but there have been some smallish issues with it, particularly with the markup generation part, and it's also doing some things in a way that just doesn't work for me – the xtemplates thing being one of these. In many ways it's less about features, and more about style.
      In MarkupMenu I've tried to correct these little hiccups, modernise the default markup, and allow for more flexibility with placeholder variables and additional / different options. MarkupMenu was built for ProcessWire 3.0.112+ and PHP 7.1+, it's installable with Composer, and I have a few additional ideas (such as conditional placeholders) on my todo list.
      One smallish and rather specific difference is that MarkupMenu supports overriding default options via $config->MarkupMenu. I find myself redefining the default markup for every site, which until now meant that each site has a wrapper function for MarkupSimpleNavigation (to avoid code / config repetition), and this way I've been able to omit that 🙂
      Requirements
      ProcessWire >= 3.0.112 PHP >= 7.1.0 If you're working on an earlier version of ProcessWire or PHP, use MarkupSimpleNavigation instead.
    • By Robin S
      Repeater Images
      Adds options to modify Repeater fields to make them convenient for "page-per-image" usage. Using a page-per-image approach allows for additional fields to be associated with each image, to record things such as photographer, date, license, links, etc.
      When Repeater Images is enabled for a Repeater field the module changes the appearance of the Repeater inputfield to be similar (but not identical) to an Images field. The collapsed view shows a thumbnail for each Repeater item, and items can be expanded for field editing.
      Screencast

      Installation
      Install the Repeater Images module.
      Setup
      Create an image field to use in the Repeater field. Recommended settings for the image field are "Maximum files allowed" set to 1 and "Formatted value" set to "Single item (null if empty)". Create a Repeater field. Add the image field to the Repeater. If you want additional fields in the Repeater create and add these also. Repeater Images configuration
      Tick the "Activate Repeater Images for this Repeater field" checkbox. In the "Image field within Repeater" dropdown select the single image field. You must save the Repeater field settings to see any newly added Image fields in the dropdown. Adjust the image thumbnail height if you want (unlike the core Images field there is no slider to change thumbnail height within Page Edit). Note: the depth option for Repeater fields is not compatible with the Repeater Images module.
      Image uploads feature
      There is a checkbox to activate image uploads. This feature allows users to quickly and easily add images to the Repeater Images field by uploading them to an adjacent "upload" field.
      To use this feature you must add the image field selected in the Repeater Images config to the template of the page containing the Repeater Images field - immediately above or below the Repeater Images field would be a good position.
      It's recommended to set the label for this field in template context to "Upload images" or similar, and set the visibility of the field to "Closed" so that it takes up less room when it's not being used. Note that when you drag images to a closed Images field it will automatically open. You don't need to worry about the "Maximum files allowed" setting because the Repeater Images module overrides this for the upload field.
      New Repeater items will be created from the images uploaded to the upload field when the page is saved. The user can add descriptions and tags to the images while they are still in the upload field and these will be retained in the Repeater items. Images are automatically deleted from the upload field when the page is saved.
      Tips
      The "Use accordion mode?" option in the Repeater field settings is useful for keeping the inputfield compact, with only one image item open for editing at a time. The "Repeater item labels" setting determines what is shown in the thumbnail overlay on hover. Example for an image field named "image": {image.basename} ({image.width}x{image.height})  
      https://github.com/Toutouwai/RepeaterImages
      https://modules.processwire.com/modules/repeater-images/
    • By EyeDentify
      Hello There Guys.

      I am in the process of getting into making my first modules for PW and i had a question for you PHP and PW gurus in here.

      I was wondering how i could use an external library, lets say TwitterOAuth in my PW module.
      Link to library
      https://twitteroauth.com/

      Would the code below be correct or how would i go about this:
      <?PHP namespace ProcessWire; /* load the TwitterOAuth library from my Module folder */ require "twitteroauth/autoload.php"; use Abraham\TwitterOAuth\TwitterOAuth; class EyeTwitter extends WireData,TwitterOAuth implements Module { /* vars */ protected $twConnection; /* extend parent TwitterOAuth contructor $connection = new TwitterOAuth(CONSUMER_KEY, CONSUMER_SECRET, $access_token, $access_token_secret); */ public function myTwitterConnection ($consumer_key, $consumer_secret, $access_token, $access_token_secret) { /* save the connection for use later */ $this->twConnection = TwitterOAuth::__construct($consumer_key, $consumer_secret, $access_token, $access_token_secret); } } ?> Am i on the right trail here or i am barking up the wrong tree?
      I don´t need a complete solution, i just wonder if i am including the external library the right way.
      If not, then give me a few hint´s and i will figure it out.

      Thanks a bunch.

      /EyeDentify
    • By dimitrios
      Hello,
      this module can publish content of a Processwire page on a Facebook page, triggered by saving the Processwire page.
      To set it up, configure the module with a Facebook app ID, secret and a Page ID. Following is additional configuration on Facebook for developers:
      Minimum Required Facebook App configuration:
      on Settings -> Basics, provide the App Domains, provide the Site URL, on Settings -> Advanced, set the API version (has been tested up to v3.3), add Product: Facebook Login, on Facebook Login -> Settings, set Client OAuth Login: Yes, set Web OAuth Login: Yes, set Enforce HTTPS: Yes, add "http://www.example.com/processwire/page/" to field Valid OAuth Redirect URIs. This module is configurable as follows:
      Templates: posts can take place only for pages with the defined templates. On/Off switch: specify a checkbox field that will not allow the post if checked. Specify a message and/or an image for the post.
      Usage
      edit the desired PW page and save; it will post right after the initial Facebook log in and permission granting. After that, an access token is kept.
       
      Download
      PW module directory: http://modules.processwire.com/modules/auto-fb-post/ Github: https://github.com/kastrind/AutoFbPost   Note: Facebook SDK for PHP is utilized.


×
×
  • Create New...