Module: Import Pages from CSV file

[dragan]

Another question: For quickly importing a list of events, is there a way to also support datetime fields? Does PW internally store these values in Unixtime?

ahem, I just need to be using the exact same date / datetime format as specified in the field definition (e.g. in my case, 2014-12-31 19:00:00).

[nyo08]

Hello,

This module works great on my PW Website... except one thing : if I import CSV file with special characters in fields (like é, è or '), it doesn't understand and import just part of the field with no special characters.

example: I have a field with : L'espagne --> import just the "L" and that's it.

How can I add the interpretation of special characters in this module ?

Thanks for your help.

[RyanJ]

Check the settings of your csv file. Make sure it is UTF-8. The module has no problem importing special characters if your csv file is saved with the correct settings. Someone has asked the question already on this topic here

[nyo08]

Check the settings of your csv file. Make sure it is UTF-8. The module has no problem importing special characters if your csv file is saved with the correct settings. Someone has asked the question already on this topic here

yes.... thanks 

[LostKobrakai]

Why isn't there a option for semicolons as seperators?

[mr-fan]

Could this module used from the api with presets for the options/csvfields?

Need such import on a user page for a file with the same data but different amount of lines...a kind of preset would be great.

(Just fast question via phone)

[LostKobrakai]

I don't think it's possible to use the module directly as it depends on form input data, but you could certainly just use the code from the module and modify it to your own extends.

[Peter Knight]

This is such a great Module. Just discovered it and imported over 1000 new pages from https://www.mockaroo.com/ in a few seconds.

[mr-fan]

Don't have much experience with really big imports it seems that the import module break on german Umlauts - all data i've imported is cutted of on every Field a öäü is in the text??

Field import "Name" => "Grötzinger" get to only  "title" => "Gr" on the page?

Or am i make a mistake with the handling of the data?

regards mr-fan

[adrian]

This is such a great Module. Just discovered it and imported over 1000 new pages from https://www.mockaroo.com/ in a few seconds.

Thanks for pointing out Mockaroo - looks really useful!

Don't have much experience with really big imports it seems that the import module break on german Umlauts - all data i've imported is cutted of on every Field a öäü is in the text??

Not sure about this module, but I just tested with the Add/Overwrite options in Batch Child Editor and pasting in CSV content with Grötzinger as the title worked fine for me. Perhaps you could see if that works for you and then maybe figure out if there is actually an issue with this module.

[SiNNuT]

Don't have much experience with really big imports it seems that the import module break on german Umlauts - all data i've imported is cutted of on every Field a öäü is in the text??

Field import "Name" => "Grötzinger" get to only "title" => "Gr" on the page?

Or am i make a mistake with the handling of the data?

regards mr-fan

It is important that you use a properly utf-8 encoded csv file so that PW understands. The characters you named should be no problem.

[kongondo]

This is such a great Module. Just discovered it and imported over 1000 new pages from https://www.mockaroo.com/ in a few seconds.

Nice one. I use this http://www.generatedata.com/. Installed locally, totally free, no row limit  (check the About tab to download the script)

Edited March 30, 2015 by kongondo

[Robin S]

A couple of minor issues I noticed when the CSV being imported contains File fields:

1. Filenames get converted to lowercase, so "MyFile.pdf" becomes "myfile.pdf".

2. There's no notification if the file doesn't exist at the path given, and a dummy file can end up being created containing the 404 HTML markup. So if the file path I'm importing is "http://mywebsite.com/tempfolder/myfile.pdf" and there is no such file at that path, a file "myfile.pdf" gets created in "/site/assets/files/..." but this file is not a valid PDF and actually contains the HTML for the 404 page.

Edit: another anomaly is that page titles are truncated if they include an en dash.

Turns out that's an issue with the CSV not being encoded as UTF-8. Good ol' Excel.

[valan]

Ryan, are there any plans to enable import for multi-language fields? Or may be there are other ways to import pages with multi-language fields in PW?

[theoretic]

Hi guys, and thank Ryan for the great job!

I would like to give more attention to this mod:

https://processwire.com/talk/topic/8788-trying-to-add-addl-fieldtype-support-to-importpagescsv-mapmarker-repeater-page-%E2%9C%93/

Currently i'm trying to import a list of pages with Map Marker field. Tried the mod, it doesn't work for me. The rest of fields are importing as expected. Noticed that the MySQL table storing my Map Marker field values stores address in the field named 'data'. Tried this in importPageValue():
 

$page->set($name->address, $value);//doesn't work!
$page->set($name->data, $value);//doesn't work!
$page->set($name->lat, $value);//doesn't work!

Hope someone may help. Thanks in advance!

[adrian]

@theoretic - welcome to the forums.

I haven't used this module for importing data into MapMarker fields, but I just tested my BatchChildEditor (https://processwire.com/talk/topic/6102-batch-child-editor/) and it is working great for MapMarker fields.

You will need to configure the following to get it to work:

Module config:

Where editing tools are available and separately configurable > Configurable Pages - choose the parent page of the children you want to create with the map marker field.

Settings tab of the parent page > Batch child editor settings:

Data entry / CSV Import Settings > CSV Field Pairings:

1:title
2:map.address
3:map.lat
4:map.lng

 

This of course should be modified as required, but it worked perfectly for me with a template that had just two fields: title, map

 

The format of the CSV I imported was:

Page One, "1 Smith St, New York, NY", 40.691049, -73.98734379999996
Page Two, "5 Brown St, Los Angeles, CA", 34.03673800000001, -118.26510000000002

Please let me know if you have any troubles getting it to work.

[theoretic]

@adrian, thank You for help!

Tried to use Your module. Compared to ImportPagesCSV it's almost rocket science but after a couple of unsuccessful attempts i managed to get the things done. Mmm, almost done. Some of the settings i tried to use:

• Mode: Update
• Field separator: tab
• Field enclosure: "
• Ignore the first row: yes
• Field pairings: 
  
  
  1:title
  2:location.address
   
• Test CSV string structure:
  
  test2 "SomeStreet, 2, building 4, City, Country, 111111"  (white space after test2 is a tab symbol)

And, after importing that string, bingo! I got the page called test2 "SomeStreet with address 2 and geolocation gone totally wrong.

Obviously it's a bug with field separator. Your module always uses comma as field separator and ignores the user-entered separator and enclosure settings. Tried also to use semicolon as separator: the issue persists. OK, i could potentially give up and use commas... but it's no way since addresses always have commas. Will be very glad to get some in-depth explanations about how to get this work.

[adrian]

Tried to ude Your module. Compared to ImportPagesCSV it's almost rocket science but after a couple of unsuccessful attempts i managed to get the things done. Mmm, almost done. Some of the settings i tried to use:

Sorry it seems like rocket science - there are definitely a lot of config options - this module is obviously about a lot more than just importing CSV files

The double quote enclosures allow you to have commas inside the address subfield as well as between the other fields - no need to use tabs as the field separators.

There does seem to be a current issue with the module making use of separators other than the default comma - I am looking into it now and should be able to fix fairly quickly. But, again if you use the quote enclosures for the address subfield this won't matter. 

I guess you aren't actually defining the lat and lng fields, but rather relying on the geolocation?

What is the 111111 at the end of the address?

[adrian]

I just pushed an update to BatchChildEditor that fixes the issue with setting a custom field separator - I made some variable name changes after my final testing for that feature and didn't change all as needed. Everything should be working on that front now if you really feel the need to use tabs.

I just tested again with a tab separated setup like in your example above and it is working fine - although I see that you got the patch for supporting MapMarker in CSVImport to work also, so you may not need my module anymore anyways. I would like to point out one advantage though - the ability to predefine csv column to field pairings so that you can make it easier for your site editors to import future addresses.

[joe_ma]

This module seems very promissing for my purpose. But I am not really sure, whether it really suits my needs. The problem is as follows:

I have exports from a filemaker data base consisting of 6 or 7 .txt files for the relational tables. Of course I can easily convert them to .csv files.

Now I should like to import these files into pw, so as the relations between the tables are kept and the data can be administrated through pw (the original filemaker data base is no longer available …). Is it possible to import these data with this module? If not, is there some other way to import the data in pw pages?

Thanks for your answers

[Robin S]

Just picking up on Ryan's earlier comment:

But if you want to get image/file descriptions in with the CSV import module, then I would probably create a temporary field to store them in, for import purposes. Then go back and paste a little code in a template to quickly convert them over...

I did this recently for pages containing multiple images and descriptions and thought I'd share the code snippet.

If your CSV data for the temporary textarea field ("desc_temp") is formatted like this...

image1.jpg=A description for image 1
image2.jpg=Another description

...etc, then after importing you can apply the descriptions with...

foreach ($mypages as $p) {
	$p->setOutputFormatting(false);
	$desc_lines = explode("\n", $p->desc_temp);
	foreach ($desc_lines as $line) {
		list($file, $desc) = explode("=", $line);
		$image = $p->images->get("name=$file");
		$image->description = "$desc"; 
	}
	$p->save();
	$p->setOutputFormatting(true);
}

[jensweigel]

Hi! I have a problem with multiple languages an importing Pages from CSV. The problem is that the other languages are not active. Are there any options to change this? All values are international (only numbers)

[LostKobrakai]

I doubt this module was created with multilanguage capabilities in mind and it won't turn other languages active by itself. Just loop over the created pages and activate them with this snippet: https://processwire-recipes.com/recipes/activate-all-languages/

[alxndre]

I'm trying to import a csv with 50,000 rows. Each page only has 2 fields - title and code - both has the same content. So far it's been running for 3 hours. It's moving but it's at a rate of 3000 per hour. Is this normal behavior? Is there a way to make this faster?

Thank you.

[kongondo]

@Alxndre'

That sounds inordinately long. Are you importing on a local or remote server? On a local server, I imported 10K rows with 3 fields in two equal batches and it took no more than 10 minutes in total (IIRC). If you can, direct SQL import will be faster than PHP. If you are working on a remote server, maybe try to import to a local server first, then get a DB dump of the import and on the remoter server import the db dump instead. Of course, page IDs will need to match what you expect on the remote server. Just a thought...