Import Pages from .csv - 500 Timeouts - Best practice import thousands of pages?

[Erik Richter]

Hey everyone!

Have just been using Ryans "Import pages from csv" module, because the UI is really nice and easy (connect fields).

I always get 500 Timeouts after about 30sec, when the csv has more than 1500 rows. I already increased memory limit and other php.ini settings - doesn't help at all. 

When I double-check after the timeout, always only about 1500 pages have been created..

Surprisingly: deleting 2500 pages via Api or admin-actions is possible and no timeouts.

So what is best practice to import 1 Million pages from a .csv ? (I already have smaller chunks of the .csv, tried with 2000, 3000 - only 1500 worked..)

Help is much appreciated ?

[Jan Romero]

Tbh for one-off imports I usually transform the csv (or whatever it is) into PHP or SQL using regex replace in Notepad++ and then just run that. 1500 rows should be nothing, unless they’re absurdly long.

edit: nvm, I missed the 1.000.000 figure, but still, writing to the database directly should be an efficient option. In MySQL might want to use LOAD DATA INFILE, but you’ll have to adjust the input data first.

[elabx]

1 hour ago, Erik Richter said:

So what is best practice to import 1 Million pages from a .csv ? (I already have smaller chunks of the .csv, tried with 2000, 3000 - only 1500 worked..)

What I've done is write a php script that I run from the command line to avoid memory/execution timeout issues. I also always use PHP League's CSV library which iterates the CSV without loading it all into memory. 

[bernhard]

See my post here:

And maybe this: 

 

[Erik Richter]

thank you everyone!!

bernhard, this last link is awesome - just tried it. great stuff, also with a nice UI. Just what I was looking for.