Module: Import Pages from CSV file

[adrian]

Hey Ryan,

Nice trick with using: $this->page->filesManager()->path();

I'll make use of that instead of the pagefileSecure check I was using in one of my modules.

Should this be listed in the cheatsheet? Seems pretty useful to me!

[RyanJ]

Since the title field creates the page name automatically and represents the url structure of the page, is there a way to define both when importing? When importing, you must declare a title field. My title fields may contain a word with an apostrophe in it.  For example, "it's" when converted to the url would be it-s. The apostrophe is replaced with a hyphen. Is there a way to just trim the apostrophe so the url would be "its"?

Thank in advance for any suggestions. 

[ryan]

Not currently with this module. Though you could write a quick file to do it for you. Place the following in your web root where ProcessWire's index.php file is, perhaps in a file called fix-names.php. Then load it in your browser. 

<?php
include("./index.php"); // bootstrap PW
$items = wire("pages")->find("template=your-template"); 
foreach($items as $item) {
  $name = preg_replace('/-s(-|$)/', '', $item->name);
  if($name != $item->name) {
    echo "<p>Changing $item->name to $name</p>";
    $item->name = $name;
    // $item->save(); 
  }
}

Uncomment that $item->save(); after you run it the first time to make sure it's doing what you want. I wrote the above here in the forum and haven't tested, so it may need tweaks. 

[RyanJ]

This is awesome Ryan! I actually used your snippet below and modified it to do what I needed and it worked perfectly, Just took me longer to create the temp field ..

foreach($mypages as $p) {
    $p->of(false);
    $p->image->description = $p->temp;
    $p->save();
}

Your above snippet is definitely a time saver and has been added to my library. It works perfectly when I tested it. I just had to modify it slightly to meet my needs.

I would just like to say your willingness to bend over backwards to assist others in this forum is a true judgement of your character. You and several others go above and beyond to help others learn with examples. One in particular I would like to mention who helped me way beyond my expectations was Nik. Just like you, he took the time to break things down with examples and even explanations. How you both manage to find the time, I do not know. To you, Nik and all the others who help us learn everyday, I thank you sincerely for helping us. I am confident I am speaking for others as well and say that ya'll truly make a difference to those of us following in your paths.

[pwFoo]

Any progress with import page (reference) fields via this module?

I need to import categories to a page reference field.

Great module!

Works fine for other fields.

[adrian]

Not exactly what you are looking for, but you could use this module: http://modules.processwire.com/modules/process-page-field-select-creator/ to create the page field and populate the categories. Might be of some help. You can paste comma separated values when entering the select options (Option 2).

[RyanJ]

Hi Xeto,

You can do this easily. Just create the categories in the backend and reference their id on the csv you are wanting to import.

See Niks post

http://processwire.com/talk/topic/383-module-import-pages-from-csv-file/?p=21476

[pwFoo]

Thank You for helping me with my problem 

It was so simple to get it work... I also found Niks post

Imported all data in two steps. First simple imported the categories and ignored all other fields. After the categories added a second import works fine! Thanks.

[Jennifer S]

Magic! Thank you, Ryan!

[PeterDK]

Hi!

I'm reading Nik's post (http://processwire.com/talk/topic/383-module-import-pages-from-csv-file/page-4#entry21476) but I cannot get my Pages to 'link'.

I'm using this line in my csv:

Ocky Docky,425,22-09-2012,,Zaterdagrit,André Ockerman,90,,,,,,,,,1306|1041|1037

The pages that I'm trying to link are 'user' templates. Selecting is absolutely ok (even works with the page's name!)

So the 'pipe' character and multi-select doesn't work :-(

Please help me out here.

[RyanJ]

You added 'FieldtypePage' to the $fieldtypes array correct? What field type does the 'user' template have? Have you tried to connect a single page id?

[PeterDK]

Goodmorning,

Yes I added the 'FieldtypePage' correctly.

After extensive testing I found out that piping id's works with the mulitple field, but not with piping the 'name' or 'title'.

The last two things work when importing a single item in this field.

So my question becomes:

How can I alter the module to support the 'title' field when multiple values are seperated by a 'pipe' (|)?

Edit:

I looked into it, and I think the core fieldtype is not adapted to my use case. Some code should be there where it's currently missing:

https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/modules/Fieldtype/FieldtypePage.module#L319

Maybe there is a quick solution to this?

[felix]

Hi Ryan,

I'm trying to import a list of retailer locations into processwire using your module.

The retailer stores all got the same name which contains an "umlaut" (ö). 

In the options I chose "Make the name unique and import new page". The Records are imported correctly but the title is cut after the umlaut (every page title is "xxx" instead of "xxxöyyy").

Do you have any suggestions how to fix this?

[RyanJ]

Hi Felix, be sure utf-8 is selected when saving your csv sheet. That should do the trick for you. 

[felix]

Doh': I thought the files already utf8 encoded. I was wrong. Shame on me!

Thanks a lot!

[bwakad]

So finally, with help of others including Joss, I managed to get a select to work...

Other people were also mentioning this module. And it's probably what I need considering I need to have select options for at least 30plus values.

But how to set up the cvs file since I only need 1 e.g. title?

[Martijn Geerts]

Every page you want to create on a new line like this:

this is a page

and this

and an other

the fourth

we continue

let's make

a few more

and again

a page

this we can use as title

etc.

...

..

.

Save this and import it.

[bwakad]

that did only one insert: the last one.

Do I need to specify each line like this:

text1,

text2,

etc?

Solved:

I had to specify the header first, like:

title

text1

text2

text3

etc

[SiNNuT]

that did only one insert: the last one.

Do I need to specify each line like this:

text1,

text2,

etc?

Yes. CSV (comma-separated values) needs a delimiter. This can be a comma, tab, semicolon or more. I haven't used this module for a while, but i think you can (or may need to) add a very first record as a header containing a list of field names. Furthermore, read the options the module gives. So the CSV something like this:

title,myfield1,myfield2
titleA,myfield1_value,myfield2_value
titleB,myfield1_value,myfield2_value
titleC,myfield1_value,myfield2_value

[formulate]

Hi Ryan (and others), thank you for this great module. I'm looking to transfer an old ModX Revo site that is currently using latin-1 encoding. My understanding from the thread here:

https://processwire.com/talk/topic/1735-get-data-from-a-latin-1-encoded-db-and-insert-it-into-pw/

is that the following code:

$newPage->title = iconv("ISO-8859-1", "UTF-8", $row['HEADLINE']);

should convert to utf-8 first. Does your import module take this in to account or will I have to somehow add this in to your module code? Or should I just somehow create a separate php file that will cycle through my CSV and do this first, then use your module to import?

I have approximately 12,000 pages of varying types to import, including user accounts.

Thanks,

K.

[horst]

Hi @formulate, welcome to the forums!

If I understand right, you first create csv-files from the iso-8859-1 site, then you import them into PW with the csv-importer module?

Why not simply convert the whole iso-files to utf-8 files before using them for the import?

------------------

There is a case study from a (lets call it 'more customized') import scenario with a W**dPr*ss site. I have done something similar with a DB from another system (Gallery2). Maybe this is of interest for you? If you do know the DB structure of the old system good enough, so that you can query the right things in the right order, you may be able to setup a single script that does the whole import in greater chunks.

Importer scripts are easy but powerfull. If you are interested we can point you to other examples or assist in building yours (if you know enough of the old system structure and how to query it).

[adrian]

Or if you have the skills, perhaps you'd be interested in creating an plug-in for Migrator, like Nico's WP migrator. We have a lot of ModX refugees here, so I am sure it would get lots of use. I don't know anything about ModX, but I'd be happy to help in any way I can.

[horst]

@adrian: good call!

[formulate]

Or if you have the skills, perhaps you'd be interested in creating an plug-in for Migrator, like Nico's WP migrator. We have a lot of ModX refugees here, so I am sure it would get lots of use. I don't know anything about ModX, but I'd be happy to help in any way I can.

While my understanding of some aspects of ModX is top-shelf, my database knowledge and experience is woeful. I can get around in phpMyAdmin and understand ModX's table structure, but writing custom SQL queries, etc. is something I've never done. Trying to create a plug-in for Migrator seems like an unachievable task for me. I think I'm going to have enough trouble as it is just trying to get my database converted properly.

Thanks for the responses.

[dragan]

This module was written before multi-language support in PW, and hasn't yet been updated to support multi language title/text/textarea fields

Would it be very difficult to create a new version supporting multi-language support?

Or are there any other tools I've missed? (i.e. I know how to create pages / populate fields via API, and also how to edit alternative languages, but it would be nice to have an automated import CSV tool for multilang sites). Perhaps a slight variation: An import tool for already existing pages, that inserts / updates one or multiple alternative language versions? If we'd declare the page ID as mandatory in the CSV, I guess this wouldn't be rocket science?