Processwire API, Loops & 504 Timeouts

[cosmicsafari]

Hi all,

Just looking for some advice.

I have created an admin page where the user can upload a fairly simple csv, in order to create a bunch of pages in batch.

I have recently been trying to update it so that it wipes the existing pages created by the last upload before creating new pages based on the newly uploaded CSV.

I'm obviously missing something as any CSV or indeed any attempt to batch import/delete always results in a 504 Timeout.

Figured this was just something we would just have to work around and likely due to server constraints.

However I recently installed the Batch Child Editor module (http://modules.processwire.com/modules/batch-child-editor) out of curiosity.

Anyway too my point, I can use this module to delete the same number or indeed import the same number of items that my implementation struggles to cope with.

So I was hoping if I posted a snippet of my own simple version, I could get some pointers as to where we think the bottleneck is as im clearly doing something wrong.

Any thoughts and advice are welcome.

<?php

if ( isset($_POST["submit"]) ) {

    if ( isset($_FILES["file"])) {

        //if there was an error uploading the file
        if ($_FILES["file"]["error"] > 0) {

            echo "Error Uploading File (Return Code: " . $_FILES["file"]["error"] . ")<br />";

        }
        else {

            $created = 0;

            //if file already exists
            if (file_exists("upload/" . $_FILES["file"]["name"])) {
                echo $_FILES["file"]["name"] . " already exists. ";
            }
            else {
                //Store file in directory "upload" with the name of "uploaded_file.txt"
                $storagename = "../assets/csv/uploaded_file.txt";
                move_uploaded_file($_FILES["file"]["tmp_name"], $storagename);

                if(($handle = fopen($storagename, 'r')) !== false)
                {
                    // get the first row, which contains the column-titles (if necessary)
                    $header = fgetcsv($handle);
                    $x=0;
                    $group_settings = wire('pages')->get('name="group-settings"');
                    // loop through the file line-by-line
                    while(($data = fgetcsv($handle)) !== false)
                    {

                        $lng = $data[0];
                        $lat = $data[1];
                        $name = $data[2];
                        $contact = $data[3];
                        $email = $data[4];
                        $phone = $data[5];


                        //CREATE
                        error_log('('.$name.') CREATE');

                        $p = new Processwire\Page();
                        $p->template = 'group';
                        $p->name = $name.'('.$lat.','.$lng.')';
                        $p->title = $name.'('.$lat.','.$lng.')';
                        $p->parent = $group_settings;
                        $p->text_a = $name;
                        $p->text_b = $contact;
                        $p->group_location->lat = $lat;
                        $p->group_location->lng = $lng;
                        $p->email = $email;
                        $p->phone_number = $phone;
                        $p->save();
                        $created++;

                        unset($data);
                    }
                    fclose($handle);
                }
            }

        }
    }
    else if ( isset($_REQUEST["delete"]) ) {

        //Delete existing locations
        $p = $pages->get('/settings/group-settings/');
        foreach($p->children as $child){
            $child->delete();
        }

    } else {

        echo "No file selected <br />";

    }
}
?>

 

 

 

 

 

[dragan]

@cosmicsafari Well, you could take a look at the BCE module...

Did you look at server logs? Did you also try with a very small CSV file - maybe it's not the size of the files, or the time it takes to process everything, but some stumbling blocks in your code ?

I would first try it with adding output formatting set to false, and then also use sanitizers (you can never trust user input), or at least trim(). I also noticed this line

$p->name = $name.'('.$lat.','.$lng.')';

which doesn't look OK. PW page names can't have special characters like (). Use the page-name $sanitizer and get rid of the () (or any other non-allowed characters).

[adrian]

1 hour ago, dragan said:

Use the page-name $sanitizer and get rid of the () (or any other non-allowed characters).

Or just set the title and leave PW to take care of the name automatically.

[cosmicsafari]

Thank you for the suggestions.

I did have a look at the content within the BCE module but found it hard to follow.

Its a fantastic module but its far more advanced than anything I have ever put together but I will continue to look into it.

Apologies I should have mentioned it does work with a small enough CSV file.

However I will make some changes and see how I get on.

I'm not sure if its relevant but the storage engine of the database is InnoDB.