Erik Richter Posted July 1, 2020 Share Posted July 1, 2020 Hey everyone! Have just been using Ryans "Import pages from csv" module, because the UI is really nice and easy (connect fields). I always get 500 Timeouts after about 30sec, when the csv has more than 1500 rows. I already increased memory limit and other php.ini settings - doesn't help at all. When I double-check after the timeout, always only about 1500 pages have been created.. Surprisingly: deleting 2500 pages via Api or admin-actions is possible and no timeouts. So what is best practice to import 1 Million pages from a .csv ? (I already have smaller chunks of the .csv, tried with 2000, 3000 - only 1500 worked..) Help is much appreciated ? Link to comment Share on other sites More sharing options...
Jan Romero Posted July 1, 2020 Share Posted July 1, 2020 Tbh for one-off imports I usually transform the csv (or whatever it is) into PHP or SQL using regex replace in Notepad++ and then just run that. 1500 rows should be nothing, unless they’re absurdly long. edit: nvm, I missed the 1.000.000 figure, but still, writing to the database directly should be an efficient option. In MySQL might want to use LOAD DATA INFILE, but you’ll have to adjust the input data first. Link to comment Share on other sites More sharing options...
elabx Posted July 1, 2020 Share Posted July 1, 2020 1 hour ago, Erik Richter said: So what is best practice to import 1 Million pages from a .csv ? (I already have smaller chunks of the .csv, tried with 2000, 3000 - only 1500 worked..) What I've done is write a php script that I run from the command line to avoid memory/execution timeout issues. I also always use PHP League's CSV library which iterates the CSV without loading it all into memory. 3 Link to comment Share on other sites More sharing options...
bernhard Posted July 1, 2020 Share Posted July 1, 2020 See my post here: And maybe this: 4 Link to comment Share on other sites More sharing options...
Erik Richter Posted July 1, 2020 Author Share Posted July 1, 2020 thank you everyone!! bernhard, this last link is awesome - just tried it. great stuff, also with a nice UI. Just what I was looking for. 1 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now