WireQueue needs testing :)

[horst]

------------------------------------------------------------------------------

This is officially released now with some fixes and a second storage handler based on SQLite3.

------------------------------------------------------------------------------

WireQueue

WireQueue is a module that allows easy creation and usage of Queues in ProcessWire.

It is based upon a basic (parent) module that can or should have one or multiple StorageHandler modules. This alpha pre release contains a basic plain text storage module, WireQueueTextfile.

The base module creates the needed:

• FIELDS (wire_queue_type, wire_queue_state, wire_queue_storage_options)
• TEMPLATES (wire-queue-container, wire-queue-storage, wire-queue-tools)
• PAGES (wire-queues = container for all Queuepages, wire-queue-storages = container for StoragetypePages)
• ROLE (wire-queue-admin)

Each storage module creates one page under wire-queue-storages.

New Queues can be created in the backend by adding a new page under "Wire Queues". After creation one have to select a Storage type from availables list and publish the page. After that, there are some buttons available to start / pause / and close the queue.

Putting and getting data to and from the queue is done via API calls. First you have to get the page that holds the queue object.

// get and validate the queue handle
if($queue = $pages->get(SELECTOR_TO_DESIRED_PAGE)->WireQueue()) {
    // wrap your data into an array and pass it into the queue
    $success = $queue->addItem($data);
    ...
}

// get and validate the queue handle
if($queue = $pages->get(SELECTOR_TO_DESIRED_PAGE)->WireQueue()) {
    $data = $queue->getItem();
    ...
}

Thats basically all what you want from a queue. Following are a few conveniences:

• $queue->getPage()->title gives you the title of the queue, ($queue->getPage() returns the page object)
• $queue->className() shows the StorageType of the queue
• $queue->getState() and $queue->getStateStr() returns the current state of a queue:
  1 = new / empty
  2 = enabled / running
  3 = paused
  4 = closed / archived
• $queue->itemCount() gives the total number of all pending items in the queue

Here is code that gives an overview of all Queues in the system:

$container = $pages->get('template=wire-queue-container');
$bodyContent = "<h1>$container->title</h1>";
$bodyContent .= "<p>There are currently {$container->numChildren} Queues defined:</p>";
$bodyContent .= "<ul>";
foreach($container->children() as $p) {
    if(! ($queue = $p->wireQueue())) continue;
    $bodyContent .= "<li>{$queue->getPage()->title}<ul>";
    if(!$queue->ready2use()) {
        $bodyContent .= "<li>{$queue->className}</li>";
        $bodyContent .= "<li>This Storagetype is not ready to use! The System seems not to provide all requirements.</li>";
        $bodyContent .= "</ul></li>";
        continue;
    }
    $bodyContent .= "<li>{$queue->className}</li>";
    $bodyContent .= "<li>{$queue->getStateStr()} ({$queue->getState()})</li>";
    $bodyContent .= "<li>Currently are {$queue->itemCount()} items pending!</li>";
    $bodyContent .= "</ul></li>";
}
$bodyContent .= "</ul>";

Following are a screenshots of the backend

The module is on Github: https://github.com/horst-n/WireQueue

Would be nice if someone has time and can test it a bit and report back.

Edited February 7, 2016 by horst
added link to forums module thread

[cstevensjr]

2016 is starting off with an enormous bang!  Great work @horst.

[horst]

I forgott to mention that it shows usefull information directly in the pagetree: total number of pending items and current states of the queues via icons

[Ivan Gretsky]

Hey, mr. horst! Looks like something I could make great usage of... But not sure that I understand the thing right. Could you please write a few words on what it is, what and how it is supposed to do. Maybe a use case...

I understand It as a way to add multiple pages through the api without the risk of running out of execution time. Am I close?

[horst]

Hey mr. Ivan! Nice to hear from you.

There can be some different usecases. Normally you want to use a Queue for decoupling time intensive tasks from the websites responses so that the users doesn't have to wait long times. For example, if someone booked / collected something on a site, there will finally an email sent to him / her. If the script would wait until it could connect to the smtp server and has sent the email, it could be that the smtp server is busy and delays the sending. With a Queue, you simply drop in this job and give fast response to the users screen: "Thanks for your booking! We will response by email soon.". Another script will pickup the job and proceed it.

So, decoupling time intensive tasks and optionally split up the work to multiple workers if usefull / necessary.

Another example would be sending a newsletter to 5k recipients. It seems better to me to have those temporary data seperated from the main Database in regard of Backups / Restores.

With WireQueue you can create and controll Queues visually in the backend. This wouldn't be necessary but I found it usefull and comfortable. Currently there is only a very simple StorageModule available with plaintext files. The next one uses SQlite Databasefiles what is much more comfortable as it opens the WireQueue to not only put and get jobs into it but also can be used to store the current state of each job. Or if the collected data should be exported to other systems for further processing, etc.

You can use whatever storage type you like. You only needs to create your own storage module upon the abstract class that comes with the module. Currently there are only comments in the code itself and the WireQueueTextfile module for detecting how it should be done. But it isn't complicated. You can setup one that uses a MYSQL table in another DB than that from your PW installation. etc. etc.

More and better information will come later in next step.

Does that help?

[Ivan Gretsky]

Sounds great. But I still can't get what a task is - is it a php script, an PW API call, a bash script... Maybe I should read some code) But if you could explain...

I have a need to import a hell load of pages with multiple images on a regular basis via cron job. Images should be processed before the import. Can I use the module for this, or is it an overkill and I should do shell scripts or something else?

[horst]

A task is something that needs to be done, e.g. sending an email at the end of someones buying in a shop. Sending the email is the "task".

You can use this module for what ever you like. It is really very basic.

If you has setup one Queue, you can access it (like in the code snippet in the first post) from a template file or from within a bootstrapscript. You need to know the name or id of the page that holds the Queue, thats all you need. Than you can pass data records to it. With the textfile storage every data record should be an array.

I don't know if this can be useful for your import script. Definitely a Pro would be if you are limited with script execution time by the server. Then you can scan / read in all needed pages / images and put them into the queue without starting to process them. After you have finished this read in, you start processing them by pulling out items from the queue as long as you are within the time limit. If you reach critical time, you stop processing and end the script. It needs to be reloaded somehow, e.g. could be done without any cron but via session->redirect() as last command that ends the script. This way you strat with a fresh amount of execution time. No record gets lost as it could be if you use only session for storage.

Maybe, you know the name Stack? It something similar as a Queue. Difference is, in a Queue is something like a pipe with both sides open. What is put in first, gets pulled out first at the other end: FiFo (First in First out). A stack is something where you put one item upon the other, maybe also a pipe, but with one end closed. Therefore you can only pull the last item from it, not the first: FiLo  (First in Last out)

WireQueue module is nothing more than the basic implementation of it into PW. ?? Ok ??

If you are unsure if this can be useful for your >cron task<, maybe you can explain a bit how this is be done. From where and when do you get the items that needs to be imported and how do you process the images and what is the next step, etc.

[horst]

Example for a use case with importing things. This is a raw draft with a bit pseudo code:

// you have created a queue in PW, the ID of the page is 2000 for example
// here is some skeleton code for an importer that uses this queue

// get and validate the queue handle
if(! ($queue = $pages->get('id=2000')->wireQueue())) exit(); // we could not get the Queue page

// now start to scan / read data for your imports, wrap each record into an array and put it into the queue
foreach($pages->find(YOURSELECTOR) as $p) {
    $data = array($p->field1, $p->field2);
    $queue->addItem($data);
}

// now start to process your items by pulling one after the other from the Queue
while($timer < $maxtime) {
    $data = $queue->getItem();
    // process the $data ...
}

$session->redirect('./');

[Frank Vèssia]

This module is amazing...I'm gonna test very soon for sending emails after user actions.

[horst]

Thanks

Testing with the plain text storage is ok in that context. But to use it for production, I would suggest you wait for the SQlite storage, because with it you will be able to store different states of an email sending job:

• new/added to the queue
• pulled by a worker for execution
• successfully sent by a worker, or not successfully sent

At least it should store sent (yes / no) and track (max) tries.

[horst]

This has got some small fixes and updates and is officially released now: https://processwire.com/talk/topic/12151-wire-queue-basic-implementation-of-simple-queues/

[Pierre-Luc]

Quick question horst, haven't had time to play with it yet but what are/will be storage options? SQLite is nice but from a distributed standpoint isn't it a bit limited ? Wouldn't MySQL be a better option? (Don't know if you support it yet, on my phone right now so limited ability to look at it)

[kongondo]

Quick question horst, haven't had time to play with it yet but what are/will be storage options? SQLite is nice but from a distributed standpoint isn't it a bit limited ? Wouldn't MySQL be a better option? (Don't know if you support it yet, on my phone right now so limited ability to look at it)

He's addressed that in the (new) official release thread found here

@horst, maybe we should now lock this topic and direct discussion to the new thread?

[horst]

This thread is closed now. All further discussions, questions moved to the release thread under this forums module section: https://processwire.com/talk/topic/12151-wire-queue-basic-implementation-of-simple-queues/