Problems with automated creation of templates and pages

[pokmot]

I need to import up to 30,000 records, and add regular updates with more records, so automation is one of the first things I am looking for. I've already created a template that displays my data nicely, and I'm pretty happy so far.

However, I experienced some problems when trying to automate importing of records. I have some code that will fail on first run, but succeed on the second run. It seems that when a new "page" is created, the system isn't keeping the correct information so the new record cannot be used immediately.

Yesterday, my "real" code failed due to duplicate 'name' - I am using code found on the forum to append '-1', '-2' etc, but on the first run the code failed to find the records just added. On the second run it worked perfectly, every time.

It therefore doesn't seem that new objects are correctly maintained in memory, and there is a mismatch between the information stored in the object and the information stored in the database.

In order to get a tight use case where it fails, I produced some concept code (attached). This code will fail on the first run, but always succeeds on the second run.

On the first run it fails when trying to set the template of a new page to the template that was created in the same run. On reloading the page, the template is loaded from the database and the attempt to create the page is successful.

The error message is:

Uncaught exception 'WireException' with message 'Invalid value sent to Page::setTemplate' in J:\wamp\www\wire\core\Page.php:593
Stack trace:
#0 J:\wamp\www\wire\core\Page.php(282): Page->setTemplate('mytemplate')
#1 J:\wamp\www\wire\core\Page.php(569): Page->set('template', 'mytemplate')
#2 J:\wamp\www\test.php(61): Page->__set('template', 'mytemplate')
#3 {main}
thrown (line 593 of J:\wamp\www\wire\core\Page.php)

I have commented out line 49 - if that line is enabled it always fails at that point. This line simply attempts to reload the "parent" holding page, and is identical to the code used in line 37, which either returns a Page or a NullPage object. Line 49 is only executed when a new holding page is added.

Also check out the "numChildren" property of the parent page as new pages are added. This remains whatever it was on the first load of the object, although several new pages are added on each load.

This makes it very difficult to automate the task (I am creating hierarchical pages, just like this example), and also means that any time an object is changed a page reload is required to ensure the correct information is available.

Or am I missing something completely obvious here? Is there a 'flush' or 'reload' option somewhere? Or a mistake in my logic for creating new templates and pages?

To test, have a 100% plain, freshly installed version of ProcessWire. I used a zip downloaded from Github today (no, didn't use GIT). I also made a backup of the database immediately after installing, and ensured I restored the database every time I needed to reset. Run the code 2-3 times by a browser refresh before resetting the database, to see the points mentioned above. I stored the file in the PW root directory, next to 'index.php'.

The attached file only contains my "test.php", executed like this: http://localhost/test.php

Thanks for any help with this.

Regards,

Steve

test.zip

[Adam Kiss]

Hm, parts of that code (I currently don't have "test shit out of it" processwire installation, and I'm lazy to do so ) ran okay for me;

Have you tried passing '$template->name' rather than '$template' as Page constructor argument?

[Soma]

just tested it on my local testversion. I experience the same effect with lates commit.

It seems the template isn't applicable. It's possible to output name and fieldgroup "$template->name" ... but a wire("templates")->get("mytemplate") seems to return NULL.

Don't know what's happening and how to fix this right now. Ryan would certainly now.

[pokmot]

Hi Soma,

Thanks for confirming there is a problem.

Did you try to uncomment line 49?

  // If the following line is enabled then execution fails here
  // Reloading the page that has just been created therefore doesn't work
  $parent = wire('pages')->get('/holding-page');

With that the request fails with an SQL error.

Adam, if you pass the name rather than the object you get an error: "Argument 1 passed to Page::__construct() must be an instance of Template, string given". Could very well be a bug in a later revision, so if you didn't use a fresh installation you might not be affected.

[Soma]

Problem is it fails creating the parent page because the template newly created isn't accessible through wire (so error is shown "Invalid value..." first time running the script). So page creation fails. After that, the template is already there, and thus works.

[ryan]

I'm not near the computer till tomorrow otherwise I'd be able to troubleshoot. The iPhone tells me it cant download a zip file . I was creating templates and assigning them to pages I'm the same request as recently as yesterday when working on the language installer. But I suspect I've not done it in the way it sounds like you are here. Make sure you have saved the template before attempting to add to a page. The way you describe it sounds like the newly created template isn't yet in the $templates var (assuming you saved it). If that's the case it would be a bug. As a temporary solution, just give the page the template object you made rather than a template name. That's more efficient anyway. It's the way I've always done it, which may be why the issue hasn't turned up earlier. But if it's a bug I can fix it tomorrow. Also there was a question that made me think $pages->uncacheAll() might help. That just clears out the current memory and selector cache, and can be helpful when you are dealing with large imports. Though PW should automatically do that when you save a page.

Edit: also, if you are dealing with 30k items you may very well run out of memory. You'll want to paginate the import or do it with multiple executions. 500-1000 page chunks is what I usually limit to, but it depends entirely on how heavy the data you are dealing with. But since you are dealing with large numbers of pages, you'll have to take more care when working with the API. For instance, $page->children() will load all children. So you need to consider limits like $page->children("limit=100"), and the like... Things that you don't usually need to think about on smaller scale use. There is a thread in the FAQ forum that covers this better. I'd link to it but I'm on a cell phone

[Soma]

here the code:

<?php

include('index.php');

print '<body style="font-family: Verdana; font-size:12px">';

function listPage($page, $level = 0) {
 echo str_repeat("    ", $level) . $page->title . "\n";
 foreach ($page->children as $child) listPage($child, $level + 1);
}

function listPages() {
 print "<pre>";
 listPage(wire('pages')->get("/")); // start at homepage
 print "</pre>";
}

listPages();

$tmpl = wire('templates')->get('mytemplate');

if (is_null($tmpl)) {
 $fieldgroup = new Fieldgroup();
 $fieldgroup->name = "mytemplate";
 $fieldgroup->add("title"); // add some fields
 $fieldgroup->add("body");
 $fieldgroup->save();

 $tmpl = new Template();
 $tmpl->name = 'mytemplate';
 $tmpl->fieldgroup = $fieldgroup;
 $tmpl->save();
 print "template created: " . $tmpl . "\n";

}

$parent = wire('pages')->get('/holding-page');
if ($parent instanceof NullPage) {
 // Create a new Parent page
 $parent = new Page();
 $parent->template = $tmpl;
 $parent->name = 'holding-page';
 $parent->title = 'Holding Page';
 $parent->body = 'This is a parent holding page';
 $parent->parent = wire('pages')->get('/');
 $parent->save();

 print "Created 'Holding Page'<br>";
 // If the following line is enabled then execution fails here
 // Reloading the page that has just been created therefore doesn't work
 //$parent = wire('pages')->get("name=holding-page");
}

$data = array(
 array('testpage', 'This is test page'),
 array('testpage', 'This is test page (duplicate of first one)'),
 array('testpagea', 'This is test page A'),
 array('testpageb', 'This is test page B'),
);

foreach ($data as $d) {
 $page = new Page();
 $page->template = $tmpl;
 $page->parent = $parent;
 $page->name = $d[0];
 $page->title = $d[0];
 $page->body = $d[1];
 try {
   print "Saving {$page->name}<br>";
   $page->save();
 } catch (Exception $e) {
   $n = 0;
   $pageName = $page->name;
   do {
     $testName = $pageName . "-" . (++$n);
     print "Checking for {$testName}<br>";
     $test = $parent->child("name=$testName, include=all");
   } while ($test->id);
   $page->name = $testName;
   $page->save();
   print "Successful: {$testName}<br>";
 }
 print "(parent now has {$parent->numChildren} children)<br><br>";
 $page = null;
}

listPages();

print "</body>";

I was kinda wrong before. The "holding-page" creation is done, but it's not callable with using wire('pages'), thus subpage creation fails. I will test Ryan suggestion...

[ryan]

Seeing that code, I believe the $pages->get("/holding-page"); selector is cached, if its returning null the second time around. Try adding $pages->uncacheAll() after saving holding page. Like I say, save() should have done it already, but just curious if that helps.

[Soma]

Already tried uncacheAll, but doesn't help.

wire("templates")->get("mytemplate"); returns NULL after template is saved.

Parent page gets created. But...

wire("pages")->get("/holding-page/"); returns WireDB mysql error after parent page is saved.

Fatal error: Uncaught exception 'WireDatabaseException' with message 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND pages.id IN(1136) GROUP BY pages.id' at line 5 SELECT false AS isLoaded, pages.templates_id AS templates_id, pages.*, pages_sortfields.sortfield, (SELECT COUNT(*) FROM pages AS children WHERE children.parent_id=pages.id) AS numChildren FROM `pages` LEFT JOIN pages_sortfields ON pages_sortfields.pages_id=pages.id WHERE pages.templates_id= AND pages.id IN(1136) GROUP BY pages.id ' in /Applications/XAMPP/xamppfiles/htdocs/pw2.ch/wire/core/Database.php:72 Stack trace: #0 /Applications/XAMPP/xamppfiles/htdocs/pw2.ch/wire/core/DatabaseQuery.php(84): Database->query(Object(DatabaseQuerySelect)) #1 /Applications/XAMPP/xamppfiles/htdocs/pw2.ch/wire/core/Pages.php(308): DatabaseQuery->execute() #2 /Applications/XAMPP/xamppfiles/htdocs/pw2.ch/wire/core/Pages.php(176): Pages->getById(Array, NULL, NULL) #3 [internal functio in /Applications/XAMPP/xamppfiles/htdocs/pw2.ch/wire/core/Database.php on line 72

So the page and template is saved and there, but not really somehow to continue creating sub pages. 1136 is the parent page id.

[ryan]

Interesting, it's definitely not cached if getting a DB error. The template thing sounds like a bug, but the page thing is perplexing. Almost wonder if were hitting a MySQL query cache here. Will be able to track it down tomorrow.

[ryan]

Darn iPhone doesn't let me scroll the code to see all of that error, but I just saw the query in my email. It's missing a template_id in the query. That's why it's failing. So the two issues seem related. And this will be easier to find now that I can see what's going on here. Thanks for these great examples and all this help testing.

[Soma]

After desperately searching for what the problem is, I resign as I don't know the core system that well let alone understand it all yet. But found that it works well if you don't try to do any of these -> wire("pages")->get()... wire("templates")->get(...); after template and pages are created saved, and just use the objects you create. So removing the last listPages, the script works.

So the listPages at the end also does not work cause it iterates through wire pages array, and somehow it doesn't work cause there is no template-/page id found. But I'm sure Ryan will find the problem and may able to solve it.

[ryan]

This was really easy to fix once I tested with the example posted. Thanks for the excellent examples to test from. The latest commit is updated to reflect the issues identified here, and test.php should now run perfectly. Please let me know if you come across any other issues.

Thanks,

Ryan

[Soma]

Awesome Ryan. Now it's working fine!  ;D

[pokmot]

Hi Ryan,

That's great; works perfectly both for my test code and for the real importing code I was working on.

Cheers,

Steve