Jump to content

Module: Import Pages from CSV file


ryan

Recommended Posts

Another question: For quickly importing a list of events, is there a way to also support datetime fields? Does PW internally store these values in Unixtime?

ahem, I just need to be using the exact same date / datetime format as specified in the field definition (e.g. in my case, 2014-12-31 19:00:00).

Link to comment
Share on other sites

  • 1 month later...

Hello,

This module works great on my PW Website... except one thing : if I import CSV file with special characters in fields (like é, è or '), it doesn't understand and import just part of the field with no special characters.

example: I have a field with : L'espagne --> import just the "L" and that's it.

How can I add the interpretation of special characters in this module ?

Thanks for your help.

Link to comment
Share on other sites

  • 4 weeks later...
  • 1 month later...

Could this module used from the api with presets for the options/csvfields?

Need such import on a user page for a file with the same data but different amount of lines...a kind of preset would be great.

(Just fast question via phone)

Link to comment
Share on other sites

Don't have much experience with really big imports it seems that the import module break on german Umlauts - all data i've imported is cutted of on every Field a öäü is in the text??

Field import "Name" => "Grötzinger" get to only  "title" => "Gr" on the page?

Or am i make a mistake with the handling of the data?

regards mr-fan

Link to comment
Share on other sites

This is such a great Module. Just discovered it and imported over 1000 new pages from https://www.mockaroo.com/ in a few seconds.

Thanks for pointing out Mockaroo - looks really useful!

Don't have much experience with really big imports it seems that the import module break on german Umlauts - all data i've imported is cutted of on every Field a öäü is in the text??

Not sure about this module, but I just tested with the Add/Overwrite options in Batch Child Editor and pasting in CSV content with Grötzinger as the title worked fine for me. Perhaps you could see if that works for you and then maybe figure out if there is actually an issue with this module.

Link to comment
Share on other sites

Don't have much experience with really big imports it seems that the import module break on german Umlauts - all data i've imported is cutted of on every Field a öäü is in the text??

Field import "Name" => "Grötzinger" get to only "title" => "Gr" on the page?

Or am i make a mistake with the handling of the data?

regards mr-fan

It is important that you use a properly utf-8 encoded csv file so that PW understands. The characters you named should be no problem.

  • Like 1
Link to comment
Share on other sites

  • 2 months later...

A couple of minor issues I noticed when the CSV being imported contains File fields:

1. Filenames get converted to lowercase, so "MyFile.pdf" becomes "myfile.pdf".

2. There's no notification if the file doesn't exist at the path given, and a dummy file can end up being created containing the 404 HTML markup. So if the file path I'm importing is "http://mywebsite.com/tempfolder/myfile.pdf" and there is no such file at that path, a file "myfile.pdf" gets created in "/site/assets/files/..." but this file is not a valid PDF and actually contains the HTML for the 404 page.

Edit: another anomaly is that page titles are truncated if they include an en dash.

Turns out that's an issue with the CSV not being encoded as UTF-8. Good ol' Excel.

Link to comment
Share on other sites

  • 3 weeks later...

Hi guys, and thank Ryan for the great job!

I would like to give more attention to this mod:

https://processwire.com/talk/topic/8788-trying-to-add-addl-fieldtype-support-to-importpagescsv-mapmarker-repeater-page-%E2%9C%93/

Currently i'm trying to import a list of pages with Map Marker field. Tried the mod, it doesn't work for me. The rest of fields are importing as expected. Noticed that the MySQL table storing my Map Marker field values stores address in the field named 'data'. Tried this in importPageValue():
 

$page->set($name->address, $value);//doesn't work!
$page->set($name->data, $value);//doesn't work!
$page->set($name->lat, $value);//doesn't work!

Hope someone may help. Thanks in advance!

Link to comment
Share on other sites

@theoretic - welcome to the forums.

I haven't used this module for importing data into MapMarker fields, but I just tested my BatchChildEditor (https://processwire.com/talk/topic/6102-batch-child-editor/) and it is working great for MapMarker fields.

You will need to configure the following to get it to work:

Module config:

Where editing tools are available and separately configurable > Configurable Pages - choose the parent page of the children you want to create with the map marker field.

Settings tab of the parent page > Batch child editor settings:

Data entry / CSV Import Settings > CSV Field Pairings:

1:title
2:map.address
3:map.lat
4:map.lng
 
This of course should be modified as required, but it worked perfectly for me with a template that had just two fields: title, map
 
The format of the CSV I imported was:
Page One, "1 Smith St, New York, NY", 40.691049, -73.98734379999996
Page Two, "5 Brown St, Los Angeles, CA", 34.03673800000001, -118.26510000000002

Please let me know if you have any troubles getting it to work.

Link to comment
Share on other sites

@adrian, thank You for help!

Tried to use Your module. Compared to ImportPagesCSV it's almost rocket science but after a couple of unsuccessful attempts i managed to get the things done. Mmm, almost done. Some of the settings i tried to use:

  • Mode: Update
  • Field separator: tab
  • Field enclosure: "
  • Ignore the first row: yes
  • Field pairings: 

    1:title
    2:location.address
     
  • Test CSV string structure:

    test2 "SomeStreet, 2, building 4, City, Country, 111111"  (white space after test2 is a tab symbol)

And, after importing that string, bingo! I got the page called test2 "SomeStreet with address 2 and geolocation gone totally wrong.

Obviously it's a bug with field separator. Your module always uses comma as field separator and ignores the user-entered separator and enclosure settings. Tried also to use semicolon as separator: the issue persists. OK, i could potentially give up and use commas... but it's no way since addresses always have commas. Will be very glad to get some in-depth explanations about how to get this work.

Link to comment
Share on other sites

Tried to ude Your module. Compared to ImportPagesCSV it's almost rocket science but after a couple of unsuccessful attempts i managed to get the things done. Mmm, almost done. Some of the settings i tried to use:

Sorry it seems like rocket science - there are definitely a lot of config options - this module is obviously about a lot more than just importing CSV files :)

The double quote enclosures allow you to have commas inside the address subfield as well as between the other fields - no need to use tabs as the field separators.

There does seem to be a current issue with the module making use of separators other than the default comma - I am looking into it now and should be able to fix fairly quickly. But, again if you use the quote enclosures for the address subfield this won't matter. 

I guess you aren't actually defining the lat and lng fields, but rather relying on the geolocation?

What is the 111111 at the end of the address?

Link to comment
Share on other sites

I just pushed an update to BatchChildEditor that fixes the issue with setting a custom field separator - I made some variable name changes after my final testing for that feature and didn't change all as needed. Everything should be working on that front now if you really feel the need to use tabs.

I just tested again with a tab separated setup like in your example above and it is working fine - although I see that you got the patch for supporting MapMarker in CSVImport to work also, so you may not need my module anymore anyways. I would like to point out one advantage though - the ability to predefine csv column to field pairings so that you can make it easier for your site editors to import future addresses.

  • Like 1
Link to comment
Share on other sites

  • 4 weeks later...

This module seems very promissing for my purpose. But I am not really sure, whether it really suits my needs. The problem is as follows:

I have exports from a filemaker data base consisting of 6 or 7 .txt files for the relational tables. Of course I can easily convert them to .csv files.

Now I should like to import these files into pw, so as the relations between the tables are kept and the data can be administrated through pw (the original filemaker data base is no longer available …). Is it possible to import these data with this module? If not, is there some other way to import the data in pw pages?

Thanks for your answers

Link to comment
Share on other sites

Just picking up on Ryan's earlier comment:

But if you want to get image/file descriptions in with the CSV import module, then I would probably create a temporary field to store them in, for import purposes. Then go back and paste a little code in a template to quickly convert them over...

I did this recently for pages containing multiple images and descriptions and thought I'd share the code snippet.

If your CSV data for the temporary textarea field ("desc_temp") is formatted like this...

image1.jpg=A description for image 1
image2.jpg=Another description

...etc, then after importing you can apply the descriptions with...

foreach ($mypages as $p) {
	$p->setOutputFormatting(false);
	$desc_lines = explode("\n", $p->desc_temp);
	foreach ($desc_lines as $line) {
		list($file, $desc) = explode("=", $line);
		$image = $p->images->get("name=$file");
		$image->description = "$desc"; 
	}
	$p->save();
	$p->setOutputFormatting(true);
}
  • Like 1
Link to comment
Share on other sites

  • 3 weeks later...
  • 1 month later...

I'm trying to import a csv with 50,000 rows. Each page only has 2 fields - title and code - both has the same content. So far it's been running for 3 hours. It's moving but it's at a rate of 3000 per hour. Is this normal behavior? Is there a way to make this faster?

Thank you. :)

Link to comment
Share on other sites

@Alxndre'

That sounds inordinately long. Are you importing on a local or remote server? On a local server, I imported 10K rows with 3 fields in two equal batches and it took no more than 10 minutes in total (IIRC). If you can, direct SQL import will be faster than PHP. If you are working on a remote server, maybe try to import to a local server first, then get a DB dump of the import and on the remoter server import the db dump instead. Of course, page IDs will need to match what you expect on the remote server. Just a thought...

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...