Executing long tasks in ProcessWire

[mtwebit]

I was looking for a module that allows the execution of long-running tasks (working with tens of thousands of pages) and could not find a suitable solution.

It started with the import problem. I have lots of data in XML form (20k+ complex entries) that I want to import into ProcessWire.
XMLReader() works fine but it takes a very long time to import all data so a simple "upload + process data on page save" would not work.

So I've created a new module for this. Meet my first (well, third) PW module: Tasker.
It's a simple module that executes long tasks in the background (using Cron or LazyCron) and reports their status to the user using a JQuery progressbar.

Any suggestions are welcome. How do you solve similar problems?

E.g. which is the best way to delete large number of pages? (max_exec_time will expire so I check it before the delete() call.)

$children = $page->children('template='.$this->template.',include=all'); // creates lot of page objects, may not have time to delete all
+ ...->delete()
OR
$childIDs = $this->pages->findIDs('parent='.$page->id.',template='.$this->template.',include=all'); // will create page objects later
+ $pages->get(..id..)->delete()

Be nice (I know you're always), I'm a PW-newbie, just started working with ProcessWire. I was developing sites with Drupal for a very long time but their non-existent module upgrade path finally has driven me away from it.

[dragan]

Congrats to your new module. I'm sure it will be a godsend for large import (or utility) tasks. I hope I'll have time to try it out soon.

As for your question: I guess I'd try to first get the IDs. And I'll probably also use findMany() - apparently it's faster than find().

https://processwire.com/api/ref/pages/find-many/

https://processwire.com/blog/posts/find-and-iterate-many-pages-at-once/

Edited November 1, 2017 by dragan
typo