Jump to content
Erik Richter

Import Pages from .csv - 500 Timeouts - Best practice import thousands of pages?

Recommended Posts

Hey everyone!

Have just been using Ryans "Import pages from csv" module, because the UI is really nice and easy (connect fields).

I always get 500 Timeouts after about 30sec, when the csv has more than 1500 rows. I already increased memory limit and other php.ini settings - doesn't help at all. 

When I double-check after the timeout, always only about 1500 pages have been created..

Surprisingly: deleting 2500 pages via Api or admin-actions is possible and no timeouts.

So what is best practice to import 1 Million pages from a .csv ? (I already have smaller chunks of the .csv, tried with 2000, 3000 - only 1500 worked..)

Help is much appreciated 🙂

Share this post


Link to post
Share on other sites

Tbh for one-off imports I usually transform the csv (or whatever it is) into PHP or SQL using regex replace in Notepad++ and then just run that. 1500 rows should be nothing, unless they’re absurdly long.

edit: nvm, I missed the 1.000.000 figure, but still, writing to the database directly should be an efficient option. In MySQL might want to use LOAD DATA INFILE, but you’ll have to adjust the input data first.

Share this post


Link to post
Share on other sites
1 hour ago, Erik Richter said:

So what is best practice to import 1 Million pages from a .csv ? (I already have smaller chunks of the .csv, tried with 2000, 3000 - only 1500 worked..)

What I've done is write a php script that I run from the command line to avoid memory/execution timeout issues. I also always use PHP League's CSV library which iterates the CSV without loading it all into memory. 

  • Like 3

Share this post


Link to post
Share on other sites

See my post here:

And maybe this: 

 

  • Like 4

Share this post


Link to post
Share on other sites

thank you everyone!!

bernhard, this last link is awesome - just tried it. great stuff, also with a nice UI. Just what I was looking for. 

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...