$page not saving during Bulk Upload

[FrancisChung]

Hi there,

I've got some code that bulk uploads the contents of Tab Delimited files into our Processwire CMS.

It looks like the $page->save functionality stops working after a certain number has been processed.

Haven't worked out where it starts failing yet but all the rows towards the bottom have certainly failed.

Unfortunately, there isn't anything in the error logs (at least the php error log) to see if there is an issue.

I have tested the rows that were not working by putting them individually in seperate tsv files to prove that the code works, the data is clean and it actually updates the CMS.

I was wondering if there's an internal limit on how many page saves the CMS can handle?

 

Thanks in advance.

 

Below is the copy of the code I'm using to do a bulk upload.

I'm using a 3rd party module called SimpleExcel to help with the reading and processing of the tab delimited files.

 

        $_rowStart = 13;

        $_colTitle  = 1; //A
        $_colMeta   = 10;//J    Meta Title = SEO Title
        $_colDesc   = 11;//K
        $_colURL    = 12;//L
        $_colURLnew = 13;//M
        $_colCanonical = 14;//N
        $_colAlt    = 15;//O
        $_colKeywords   = 16;//P

        $excel = new SimpleExcel('TSV');
        $excel->parser->loadFile(__DIR__."/{$file}");

        //$url =  $excel->parser->getCell($_rowStart, $_colURL );
        $row= $_rowStart;
        do
        {
            $title  =   $excel->parser->getCell($row, $_colTitle );
            $meta   =   $excel->parser->getCell($row, $_colMeta );
            $desc   =   $excel->parser->getCell($row, $_colDesc );
            $url    =   $excel->parser->getCell($row, $_colURL );
            $urlnew =   $excel->parser->getCell($row, $_colURLnew );
            $canonical= $excel->parser->getCell($row, $_colCanonical );
            $alt    =   $excel->parser->getCell($row, $_colAlt );
            $keywords=  $excel->parser->getCell($row, $_colKeywords );

            $internal=  str_replace("http://sprachspielspass.de",'',$url);

            $page =  wire(pages)->get("path=$internal");

            //TODO: Error handling and logging
            if (!IsNullPage($page))
            {
                $page->setOutputFormatting(false);

                $page->title = $title;
                $page->seo_title = $meta;
                $page->seo_description = $desc;
                $page->url = $urlnew;
                $page->seo_canonical = $canonical;
                $page->image_alt = $alt;
                $page->seo_keywords = $keywords;

                $page->Save();

                WriteLog("{$internal} saved");
            }
            else
            {
                WriteLog("{$internal} NOT FOUND!");
            }

            $row++;

        }  while (!IsNullOrEmptyString($url));

Edited December 8, 2015 by LostKobrakai
Please use the code blocks the forum does provide

[BitPoet]

Are you calling this from the command line or via http? In the latter case, my guess would go towards script execution time constraints.

Are you getting a "saved" message for those failed pages?

I don't think there's a limit for the number of page saves. I know I've imported a few thousand pages in one go in my last project, with each import resulting in more than two page saves.

[FrancisChung]

I've tried setting ini_set('max_execution_time', 600);

Unfortunately it hasn't fixed my issue.

I will try and add more debugging info into the code and try and get to the bottom of it.

I did a rough count of the rows and the rows in logs does not match the rows in the tsv files.

I will have to take a closer look.

Thanks for the heads up about the execution script time and the row count

[kongondo]

Try to set max exec time within the loop itself, inside your if block
 

$page->setOutputFormatting(false);
set_time_limit(60); 

 
If memory is a problem, you might also want to free some memory by adding the following after WriteLog.

WriteLog("{$internal} saved");
wire('pages')->uncacheAll(); // free some memory 

[FrancisChung]

Unfortunately, it seems to be an encoding issue.

The file was encoded in UTF8-BOM which has been fine reading German Umlaut characters up until now.

But it seems the newest entries are returning blanks when there's entries in the tab delimited file. 

[kongondo]

Does this help? I use it myself in my Matrix module. 

[FrancisChung]

It turned out to be a line break in some of the rows which I couldn't spot with the various text editors I was using .... Aaargh .... I'll probably have to strip them out before I process the file again.

Again, thanks for the prompt responses.

You guys are the best!