Jump to content

$page not saving during Bulk Upload


FrancisChung
 Share

Recommended Posts

Hi there,

I've got some code that bulk uploads the contents of Tab Delimited files into our Processwire CMS.

It looks like the $page->save functionality stops working after a certain number has been processed.

Haven't worked out where it starts failing yet but all the rows towards the bottom have certainly failed.

Unfortunately, there isn't anything in the error logs (at least the php error log) to see if there is an issue.

I have tested the rows that were not working by putting them individually in seperate tsv files to prove that the code works, the data is clean and it actually updates the CMS.

I was wondering if there's an internal limit on how many page saves the CMS can handle?
 
Thanks in advance.
 
Below is the copy of the code I'm using to do a bulk upload.
I'm using a 3rd party module called SimpleExcel to help with the reading and processing of the tab delimited files.
 
        $_rowStart = 13;

        $_colTitle  = 1; //A
        $_colMeta   = 10;//J    Meta Title = SEO Title
        $_colDesc   = 11;//K
        $_colURL    = 12;//L
        $_colURLnew = 13;//M
        $_colCanonical = 14;//N
        $_colAlt    = 15;//O
        $_colKeywords   = 16;//P

        $excel = new SimpleExcel('TSV');
        $excel->parser->loadFile(__DIR__."/{$file}");

        //$url =  $excel->parser->getCell($_rowStart, $_colURL );
        $row= $_rowStart;
        do
        {
            $title  =   $excel->parser->getCell($row, $_colTitle );
            $meta   =   $excel->parser->getCell($row, $_colMeta );
            $desc   =   $excel->parser->getCell($row, $_colDesc );
            $url    =   $excel->parser->getCell($row, $_colURL );
            $urlnew =   $excel->parser->getCell($row, $_colURLnew );
            $canonical= $excel->parser->getCell($row, $_colCanonical );
            $alt    =   $excel->parser->getCell($row, $_colAlt );
            $keywords=  $excel->parser->getCell($row, $_colKeywords );

            $internal=  str_replace("http://sprachspielspass.de",'',$url);

            $page =  wire(pages)->get("path=$internal");

            //TODO: Error handling and logging
            if (!IsNullPage($page))
            {
                $page->setOutputFormatting(false);

                $page->title = $title;
                $page->seo_title = $meta;
                $page->seo_description = $desc;
                $page->url = $urlnew;
                $page->seo_canonical = $canonical;
                $page->image_alt = $alt;
                $page->seo_keywords = $keywords;

                $page->Save();

                WriteLog("{$internal} saved");
            }
            else
            {
                WriteLog("{$internal} NOT FOUND!");
            }

            $row++;

        }  while (!IsNullOrEmptyString($url));
Edited by LostKobrakai
Please use the code blocks the forum does provide
Link to comment
Share on other sites

Are you calling this from the command line or via http? In the latter case, my guess would go towards script execution time constraints.

Are you getting a "saved" message for those failed pages?

I don't think there's a limit for the number of page saves. I know I've imported a few thousand pages in one go in my last project, with each import resulting in more than two page saves.

  • Like 3
Link to comment
Share on other sites

I've tried setting ini_set('max_execution_time', 600);

Unfortunately it hasn't fixed my issue.

I will try and add more debugging info into the code and try and get to the bottom of it.

I did a rough count of the rows and the rows in logs does not match the rows in the tsv files.

I will have to take a closer look.

Thanks for the heads up about the execution script time and the row count

Link to comment
Share on other sites

Try to set max exec time within the loop itself, inside your if block
 

$page->setOutputFormatting(false);
set_time_limit(60); 

 
If memory is a problem, you might also want to free some memory by adding the following after WriteLog.

WriteLog("{$internal} saved");
wire('pages')->uncacheAll(); // free some memory 
  • Like 2
Link to comment
Share on other sites

It turned out to be a line break in some of the rows which I couldn't spot with the various text editors I was using .... Aaargh .... I'll probably have to strip them out before I process the file again.

Again, thanks for the prompt responses.

You guys are the best!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...