Jump to content
horst

Wire Queue, basic implementation of simple queues

Recommended Posts

Wire Queue

Wire Queue is a module that allows easy creation and usage of Queues in ProcessWire.

It is based upon a basic parent module (WireQueue) that should have one or multiple StorageHandler modules installed too. This beta release contains a simple plain text storage module, WireQueueTextfile, and a simple Sqlite3-DB storage module, WireQueueSqlite3.

The base module creates the needed:

  • FIELDS (wire_queue_type, wire_queue_state, wire_queue_storage_options)
  • TEMPLATES (wire-queue-container, wire-queue-storage, wire-queue-tools)
  • PAGES (wire-queues = container for all Queuepages, wire-queue-storages = container for StoragetypePages)
  • ROLE (wire-queue-admin)

Each storage module creates one page under wire-queue-storages.

New Queues can be created in the backend by adding a new page under "Wire Queues". After creation one have to select a Storage type from availables list and publish the page. After that, there are some buttons available to start / pause / and close the queue.

Putting and getting data to and from the queue is done via API calls. First you have to get the page that holds the queue object.

// get and validate the queue handle
if($queue = $pages->get(SELECTOR_TO_DESIRED_PAGE)->wireQueue()) {
    // wrap your data into an array and pass it into the queue
    $success = $queue->addItem($data);
    ...
}
// get and validate the queue handle
if($queue = $pages->get(SELECTOR_TO_DESIRED_PAGE)->wireQueue()) {
    $data = $queue->getItem();
    ...
}

Thats basically all what you want from a queue. Following are a few conveniences:

  • $queue->getPage()->title gives you the title of the queue, ($queue->getPage() returns the page object)
  • $queue->className() shows the StorageType of the queue
  • $queue->getState() and $queue->getStateStr() returns the current state of a queue:
    1 = new / empty
    2 = enabled / running
    3 = paused
    4 = closed / archived
  • $queue->itemCount() gives the total number of all pending items in the queue

Here is code that gives an overview of all Queues in the system:

$container = $pages->get('template=wire-queue-container');
$bodyContent = "<h1>$container->title</h1>";
$bodyContent .= "<p>There are currently {$container->numChildren} Queues defined:</p>";
$bodyContent .= "<ul>";
foreach($container->children() as $p) {
    if(! ($queue = $p->wireQueue())) continue;
    $bodyContent .= "<li>{$queue->getPage()->title}<ul>";
    if(!$queue->ready2use()) {
        $bodyContent .= "<li>{$queue->className}</li>";
        $bodyContent .= "<li>This Storagetype is not ready to use! The System seems not to provide all requirements.</li>";
        $bodyContent .= "</ul></li>";
        continue;
    }
    $bodyContent .= "<li>{$queue->className}</li>";
    $bodyContent .= "<li>{$queue->getStateStr()} ({$queue->getState()})</li>";
    $bodyContent .= "<li>Currently are {$queue->itemCount()} items pending!</li>";
    $bodyContent .= "</ul></li>";
}
$bodyContent .= "</ul>";

Following is a screenshot of the backend

wire-queue-01.gif

The module is available in the modules directory: http://modules.processwire.com/modules/wire-queue/
Or you get it from Github: https://github.com/horst-n/WireQueue.
.
.
.
The Sqlite3 storage handler not only let you push and pull data items to and from it, it also can keep track of the current state of a record. If you use multiple / different workers for pulling and processing the data, you can store an ID for them too.
 
This is how the DB Table looks like:
pw_wirequeue_sqlite3_db-screen.jpg
 
The Wire Queue Sqlite3 storage handler provides the methods

  • addItem($arrayData)      // same as WireQueueTextfile
  • getItem($worker = null)   // same as WireQueueTextfile, (but the textfile storage cannot support $worker!)
  • updateItemState($id, $state, $worker = null)  // this you can use for further processing, if you want

 .
addItem($arrayData)
 
$arrayData is a, maybe associative, array containing all your data for one record. The method stores it under the next free id under data in the Sqlite-DB-file and sets the state to 0. The field worker is empty for new added records.

Following is a basic working example for pushing data records into a queue:

// you have created a queue in PW, the ID of the page is 1420 for example
// here is some skeleton code for an importer that uses this queue

// get and validate the queue handle
if(! ($queue = $pages->get('id=1420')->wireQueue())) exit(); // we could not get the Queuepage

// now start to scan / read data for your imports, wrap each record into an array and put it into the queue
foreach($pages->find(YOURSELECTOR) as $p) {
    $data = array($p->field1, $p->field2);
    $queue->addItem($data);
}

.

.

getItem($worker = null)
 
$worker is an integer that you can define to suite your needs. If you don't use or don't want identify multiple workers, just ommit it. The method pulls one pending record from the queue, changes the state from 0 to 1, and returns an associative array with the keys id and data. array('id' => integer, 'data' => array)
You will need the id if you further want to use the queue to keep track of processing steps.
You must pull your stored $data from $array['data'] and use the id for further storing the state.

.

updateItemState($id, $state, $worker = null)
 
$id identifys the record a worker has processed, for $state you can define and use every integer you like, but not 0 or 1. If you also want to store altered data, and not only different states, you can use updateItem($id, $state, $worker = null, $data = null) instead of updateItemState().

.

Here is a working example with a bit pseudo code how workers can get a pending record to process it further and store back the result of the process:

// you have created a queue in PW, the ID of the page is 1420 for example
// here is some skeleton code for an importer that uses this queue

// get and validate the queue handle
if(! ($queue = $pages->get('id=1420')->wireQueue())) exit(); // we could not get the Queuepage

// we do not use different workers here in that example, so we do not pass an worker id here
$tmp = $queue->getItem();   // get a record from the queue
$record_id = $tmp['id'];    // store the record id
$data = $tmp['data'];       // get the data array

// process the $data ... 
// and get the result of the process, in this example true or false
$result = processMyRecord($data);

// as new records have a state = 0, fetched records have a state = 1, 
// you may define a state of 2 for successful processed records, and higher ones for failed records
$state = true === $result ? 2 : 3;
$queue->updateItemState($record_id, $state);

.

.

getItem4FurtherProcessing($state, $worker = null)
 
The $state you pass to the method is the state you want get the record for. If there is one pending, its state will be set +1 and the id and data is passed back to you in an associative array: array('id' => integer, 'data' => array).

---------

Here is a pseudo code example how (multiple) worker scripts may batch process queue records with the sqlite storage handler

// on the server in this example, everyscript will timeout / die after 180 seconds
// we start a timer
$time = Debug::timer();

// we use different instances of workers that pull and process records from the queue, 
// so additionally to the processings states, we also want to store the worker ids
$worker = 2000;

// now start to process items by pulling one after the other from the queue
while(150 > Debug::timer($time)) {
    $tmp = $queue->getItem($worker);   // get a record from the queue
    if(!$tmp) continue;                // maybe currently are no pending records available
    $record_id = $tmp['id'];           // store the record id
    $data = $tmp['data'];              // get the data array
    $result = processMyRecord($data);  // process the data and get back the result
    $state = true === $result ? 2 : 3; // define an integer for the new state, according to the result of processing the data
    $queue->updateItemState($record_id, $state, $worker);
}

// we are close to the timeout for the script, so finish it with calling it itself to get a fresh run
$session->redirect('./');
  • Like 10

Share this post


Link to post
Share on other sites

Just to clarify: the further processing does not have to be done from within Processwire, nor does it to be done by PHP. You can use PW to only collect the data, maybe in the example Sqlite storage handler. To further process the data records, you can use every software that can communicates to a Sqlite DBfile.

If you need different fields in a Sqlite table, just copy the WireQueueSqlite3 module, - change the file- and class name to be no duplicates of the distributed storage handler, - and, at least, change the fielddefinitions for the DB table. But you can change every thing you want.

If you want or need to use other storage types and setup a new WireQueueSomething module, - please feel free to submit / commit it. :)

Share this post


Link to post
Share on other sites

hi horst,

thank you for your module. could be that i need it one day, though i have to say it was all new to me and your explanation in the other thread was helpful for me: https://processwire.com/talk/topic/11927-wirequeue-needs-testing/?p=111287 :)

thank you also for the screencasts that helped me a lot understanding what it is all about.

the further processing must not be done from within Processwire...

i think you mean "does not have to be done" as "must not be done" = "darf nicht innerhalb pw gemacht werden", right?

could you please explain why you used TXT/SQLite for data storage? why not use a mysql table from within pw? thank you :)

Share this post


Link to post
Share on other sites

@BernhardB: thanks for the language explanation, I have corrected it above in my post! :)

I haven't used MySQL tables, because my main motivation was to have a storage that not belongs to the PW-DB. I think this is useful in regard of Backup / Restore processes, if one have not mass temporary records with it. Filebased storage is portable and easy to use (TXT). SQLite is also portable but has DB support what lets you simply push, pull and update records. For example, think of sending newsletters to 5k recipients, SQLite is very convenient to handle this.

Another usecase what I have had: A website collected infos over a time of two weeks and the (post) processing should be done afterwards. Here we simply have implemented a download link for the customer into the editpage, so that he could store the textfile (CSV style), locally on his computer for further processing with excel.

So, I believe there will be also usecases where a MySQL table is a better storage. For all storage types that will need initial config params, I have implemented a simple Textareafield into the queue page, what can be used to take that settings. (one can use it as key = value pairs each in its own row, like in some other modules, maybe. But it is up to the developer if he like to use that simple approach or if he implements something other). The Textareafield is collapsed by default for admins, and hidden for others. (Whereas the queue pages require minimum the role "wire-queue-admin" or superuser for everything else than viewing the state or count).

So, hopefully someone will make a mysql storage handler and commit it here or at github. :)

(or another DB)

Edited by horst
  • Like 1

Share this post


Link to post
Share on other sites

Hey Horst, 

Looks really impressive, but I'm wondering how this module differs from the queuing done in IftRunner combined with Page Actions. I'm comparing both at the moment for a rather large project we are working on. One thing I see it that you have different queues whereas IftRunner only has one queue to store them all. Any other differences that you are aware of?

Share this post


Link to post
Share on other sites

Hi Arjen,

I haven't used IftRunner and doesn't know it. So, sorry, I cannot answer anything in regard of comparing. If you already know IftRunner and it does support what you need, I would stick with it. WireQueue is new and not (widely) tested currently. In fact, currently it is only tested by me in a few use cases. Before using it in a huge project, you definetly should do thouroughful testing.

With a quick look through the repository, it seems that queuing in IftRunner is done internally, means, it is mapped to page actions, you do not have to populate it manually, but you also cannot use it externally, without page actions. But not really sure on this, - it was my first read here.

Share this post


Link to post
Share on other sites

Thanks for responding. We are going to test it thoroughly. That's why I'm looking for all options. Pity you haven't used it yet. IftRunner with (Page) Actions are really simple as a concept, but quite powerful. Antti wrote about their use case a while ago.

  • Like 2

Share this post


Link to post
Share on other sites

...  Pity you haven't used it yet. IftRunner with (Page) Actions are really simple as a concept, but quite powerful. Antti wrote about their use case a while ago.

Thanks for the link to Anttis writeup.

------------------

Following is a description for what the SQLite handler was used in the past before I built WireQueue around it.

One use case was for an award winning Cinemaportal where they send up to 2x 13k emails in different newsletters every week. I should embed the WireMailSMTP module there and I suggested that we should use SQLite files for it.

This is running a half year in production now without any problems. At the beginning we have had a testing for 2 weeks where we slightly modified and tweeked some little parts. On that server where the website is hosted were running many many different programs and services, so, that for unknown reasons, sometimes the SQLite DB stopped working unexpected. But SQLite has a sort of transaction feature (Sqlite-Journal files), so that no recipient got lost. On every run we retrieve 500 pending recipients.

The only thing what could happen there was, that, when immediately after successful sending a mail and the SQLite crashed before it could update the record in the dbfile AND in the journal file, this single recipient will be picked up again with the next batch, what results in sending him the newsletter twice. So, this (only) happened a few times per week, (3-10) on 10k mails, = 0,1 percent. And as this may be a bit specific to that explicit server, as I haven't encountered it with other use cases, it seems to be much more secure to use SQLite with its transaction / rollback functionality as MySQL with MyISAM! Don't know about the differences when using MySQL and InnoDB. But we / you can build storage handlers for every DB we / you like. :)

So, what I have done with building WireQueue, is implementing it for easy usage and monitoring in PWs backend. post-1041-0-54734100-1455109217_thumb.pn

The previous used fieldsdefinition of the SQLite table has changed from userid, email, sent, tries to data, state, worker, but everything else of the SQLite part is identical in WireQueue.

Hope this helps a bit.

Edited by horst
added info about use cases for SQLite
  • Like 3

Share this post


Link to post
Share on other sites

hi horst,

doing first tests with your module: how can i re-open a queue after closing/archiving it?

thank you

Share this post


Link to post
Share on other sites

With the UI, you cannot! Archiveing means what it says. :)

If you want to stop using it, with the possibility to reuse it later, please use pause.

I'm not sure on this, as it isn't meant to do this by design, but you may try to cahnge the fieldvalue by API:

// get the desired queue page by its id
$queue = $pages->get("id=1234");

// get the fieldname that stores the state
$fieldname = WireQueue::WIRE_QUEUE_FIELD2;

// change value from 4 (archived) to 3 (paused), or 2 (active)
$queue->$fieldname = 3; // or 2
$queue->save();

BUT: Without warranty. I'm not sure if this has any sideeffects, its not tested!

Share this post


Link to post
Share on other sites
$pages->get(SELECTOR_TO_DESIRED_PAGE)->WireQueue() 

You have typos in your first two examples: it should be wireQueue and not WireQueue

ok no problem, but i think there should at least be a warning that this is an irreversible action :)

Share this post


Link to post
Share on other sites
$pages->get(SELECTOR_TO_DESIRED_PAGE)->WireQueue() 

You have typos in your first two examples: it should be wireQueue and not WireQueue

ok no problem, but i think there should at least be a warning that this is an irreversible action :)

Thanks for pointing to the typos. I corrected it.

  • Like 1

Share this post


Link to post
Share on other sites

hi horst,

i need a little more help, sorry. i don't really get the concept. for example when i have a queue to resize images, how would i setup this? when i have a queue "resizeimages" i could add all resize requests to this queue and then run the resize for every item. so far so good, but what if i have different users with images from different pages? user A adds 100 images to queue "resizeimages" and user B adds 100 images to the same queue. the queue now has 200 images in it if that happens at the same time. how does the queue know which images belong to wich user?

the "worker" is empty for new entries so i can't use this. or should i do an additem() and right after that an updateitemstate(...worker...)?

or am i completely on the wrong track with what i want to do?

thank you

edit: ok i don't think that would work because there would still be all items in the queue and there's no kind of "selector" to get only the items from one worker, right? so would i have to create a queue for every page where i want to resize images? i didn't try it so far, but i think that sould be possible via standard api creating new pages and setting the appropriate values?

Share this post


Link to post
Share on other sites

Sorry, I cannot follow. A queue only stores the data you put into it, nothing more. If you need references to whatever, put it into your data. That's the only way to have your needed data together. But this has nothing to do with the queue. Once again, the queue is nothing else than a storage, a DB table, with exactly 1 field for your data. Therefore, (I think you do this already), you have to wrap your data into an array. If you need more data-information as you currently have in it, just put it into the data array too. :)

Share this post


Link to post
Share on other sites

hmmm, ok i see you really didn't get me :) i was writing a more detailed answer but while i was writing i found a better solution for my usecase. i think i wanted to use the queue in a way it is not built for (but not the way you understood me, but thats no factor any more) ;)

Share this post


Link to post
Share on other sites

hi horst, me again ;)

just came across https://processwire.com/api/variables/log/ and as i read the docs i thought it is quite similar to txt-queues. i think one could easily create simple queues this way and would be interested in your opinion.

creating a new queue would be as simple as $log->save('examplequeue', 'mydata');

fetching lines from the end of the file is also easy (http://kongondo.github.io/ProcessWireAPIGen/master/source-class-WireLog.html#196-229) and could maybe modified by ryan with an option go get lines from the beginning of the file and not only the end. this functionality is already built into the filelog class http://kongondo.github.io/ProcessWireAPIGen/master/source-class-FileLog.html#164-216

only thing missing would be a method to delete rows from the beginning or end of the file, but that would also be really easy.

edit: don't know about the "workers", didn't get the concept...

Share this post


Link to post
Share on other sites

edit: don't know about the "workers", didn't get the concept...

Workers are separate processes, which run alongside your webserver stack on the host system. They can process things they get out of the queue without being reliant on the Request->Process->Response cycle of the webserver.

E.g. You want to have a way for users to request a big report on your website. As a big report sounds like, it take some time to generate the pdf for it (say ~1 Min). You wouldn't want your user to sit around and watch the nice browser spinner, waiting for the next page to pop up. You can now write the request in a queue and instantly return a message stating that the report will be generated and emailed to the user when ready.

To generate that report you may know the option of using a cron to trigger website logic, without an actual user requesting a website, but that's also flawed with things like timeouts and such, also it's only starting in intervals. That's where workers come in.

These are php (and other script) processes, which often run as services like e.g. your apache server does. These scripts contain some way to not exit early (e.g. infinite while loop) – which is the biggest difference to website request – and periodically check their queue if there's work to do. If your report request is in it, they generate the pdf, email it and if done delete the queue item. If there are other reports to do they'll move on to them, otherwise they sleep until another queue item is available. 

  • Like 5

Share this post


Link to post
Share on other sites

@horst

Thanks for this module, I've been hammering it pretty hard for the past month and have just created a pull-request over on github for your consideration. I've been having a few reliability problems with the textfile driver's itemCount() method. I also wanted a fast way to purge a queue without having to call getItem() repeatedly.

The itemCount() method sometimes returns an incorrect count (anywhere from 0 to the true number of items in the queue). This happens even when there are multiple thousands of jobs queued up and it makes it unreliable as part of a conditional statement. The explanation is given in the commit that fixes the issue as part of the pull request.

Thanks for looking!

  • Like 6

Share this post


Link to post
Share on other sites

Many thanks @netcarver.

I must admit that I have written the Textfile-driver only for this module to have a basic driver. Other as the SQLite-driver, it has not have much testing. I'm glad to hear that you have done this and you contribute to the module. I have you added as colaborator to it, so that you can change / add other improvements / enhancements as you like. ^-^;)

  • Like 3

Share this post


Link to post
Share on other sites

Thanks for adding me as a colaborator, @horst.

I've pushed a new dev branch to the repository that contains,

  • Bugfixes to the textfile driver's itemCount() method and small cleanups for edge cases.
  • New methods isEmpty(), purgeItems() and getItems() added to the base storage class and fully implemented in the textfile driver. The SQLite driver currently has stub implementations of these that need filling out.
  • The addition of a Redis driver for super-fast in-memory queues (implements the full base class interface.)

If anyone wants to give this a try before horst gets around to merging into the master branch, feel free to grab the zip (repository here.)

  • Like 5

Share this post


Link to post
Share on other sites

Horst and I would like to announce the release of v1.0.0 of the WireQueue suite of modules.

There are now drivers for textfiles, redis and SQLite and the interface has been extended to include the isEmpty(), purgeItems() and getItems() methods.

Enjoy!

  • Like 12

Share this post


Link to post
Share on other sites

I think I may have found a small problem with WireQueueTextfile, although it could be due to this PHP bug?

The array is being serialized in the addItem() function, but in my case an element's value is sometimes a string containing newline characters "\r\n". I believe WireQueueTextfile saves each array to one line, but these characters are causing the serialized string to be split onto multiple lines, which I'm assuming causes problems for WireQueue.

Would it make sense to somehow sanitize these values (outside of WireQueue) before adding them to the queue?

Share this post


Link to post
Share on other sites

Could you try replacing addItem() and getItem() with the following...

    public function addItem($arrayData) {
        if(!$this->_addItem()) return false;
        if(2 != $this->getState()) return false;
        if(!$fp = @fopen($this->getFilename(), 'ab')) return false;
        if(flock($fp, LOCK_EX)) {
            $arrayData = serialize($arrayData);
            $data = str_replace("\r\n", '\r\n', $arrayData) . "\n";
            $res = fwrite($fp, $data);
            fflush($fp);
            flock($fp, LOCK_UN);
            fclose($fp);
            return $res == strlen($data);
        }
        fclose($fp);
        return false;
    }

    public function getItem($worker = null) {
        if(!$this->_getItem()) return false;
        if(2 != $this->getState()) return false;
        if(!$fp = @fopen($this->getFilename(), 'rb+')) return false;
        if(flock($fp, LOCK_EX)) {
            $line = trim(fgets($fp));
            if(!$line) {
                flock($fp, LOCK_UN);
                fclose($fp);
                if(0 == $this->itemCount()) return null;
                return false;
            }
            // we have the first entry, now write all following data into a buffer
            $fpTmp = @fopen('php://temp/maxmemory:' . intval(1024 * 1024 * 5), 'rb+');
            while(!feof($fp)) fwrite($fpTmp, fread($fp, 4096));
            fseek($fp, 0, SEEK_SET);
            ftruncate($fp, 0);
            fseek($fpTmp, 0, SEEK_SET);
            // write back buffer into file
            while(!feof($fpTmp)) fwrite($fp, fread($fpTmp, 4096));
            fclose($fpTmp);
            fflush($fp);
            flock($fp, LOCK_UN);
            fclose($fp);
        }
        return unserialize(str_replace('\r\n', "\r\n", $line));
    }

Untested - YMMV!

  • Like 2

Share this post


Link to post
Share on other sites

No luck with that unfortunately, still lots of fragmented lines. Do the instances of '\r\n'  need to be wrapped in double quotes? I can't work out the logic here!

I should also note that these strings can include \r\n but are also sometimes littered with instances of \r or \n individually.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Robin S
      I have had this module sitting in a 95% complete state for a while now and have finally made the push to get it out there. Thanks to @teppo for his Hanna Code Helper module which I referred to and borrowed from during development.
      http://modules.processwire.com/modules/hanna-code-dialog/
      https://github.com/Toutouwai/HannaCodeDialog
      HannaCodeDialog
      Provides a number of enhancements for working with Hanna Code tags in CKEditor. The main enhancement is that Hanna tags in a CKEditor field may be double-clicked to edit their attributes using core ProcessWire inputfields in a modal dialog.
      Requires the Hanna Code module and >= ProcessWire v3.0.0.
      Installation
      Install the HannaCodeDialog module using any of the normal methods.
      For any CKEditor field where you want the "Insert Hanna tag" dropdown menu to appear in the CKEditor toolbar, visit the field settings and add "HannaDropdown" to the "CKEditor Toolbar" settings field.
      Module configuration
      Visit the module configuration screen to set any of the following:
      Exclude prefix: Hanna tags named with this prefix will not appear in the CKEditor toolbar dropdown menu for Hanna tag insertion. Exclude Hanna tags: Hanna tags selected here will not appear in the CKEditor toolbar dropdown menu for Hanna tag insertion. Background colour of tag widgets: you can customise the background colour used for Hanna tags in CKEditor if you like. Dialog width: in pixels Dialog height: in pixels Features
      Insert tag from toolbar dropdown menu
      Place the cursor in the CKEditor window where you want to insert your Hanna tag, then select the tag from the "Insert Hanna tag" dropdown.

      Advanced: if you want to control which tags appear in the dropdown on particular pages or templates you can hook HannaCodeDialog::getDropdownTags. See the forum support thread for examples .
      Edit tag attributes in modal dialog
      Insert a tag using the dropdown or double-click an existing tag in the CKEditor window to edit the tag attributes in a modal dialog.

      Tags are widgets
      Hanna tags that have been inserted in a CKEditor window are "widgets" - they have a background colour for easy identification, are protected from accidental editing, and can be moved within the text by drag-and-drop.

      Options for tag attributes may be defined
      You can define options for a tag attribute so that editors must choose an option rather than type text. This is useful for when only certain strings are valid for an attribute and also has the benefit of avoiding typos.
      Add a new attribute for the Hanna tag, named the same as the existing attribute you want to add options for, followed by "__options". The options themselves are defined as a string, using a pipe character as a delimiter between options. Example for an existing attribute named "vegetables":
      vegetables__options=Spinach|Pumpkin|Celery|Tomato|Brussels Sprout|Potato You can define a default for an attribute as normal. Use a pipe delimiter if defining multiple options as the default, for example:
      vegetables=Tomato|Potato Dynamic options
      Besides defining static options as above, you can use one Hanna tag to dynamically generate options for another. For instance, you could create a Hanna tag that generates options based on images that have been uploaded to the page, or the titles of children of the page.
      Your Hanna tag that generates the options should echo a string of options delimited by pipe characters (i.e. the same format as a static options string).
      You will probably want to name the Hanna tag that generates the options so that it starts with an underscore (or whatever prefix you have configured as the "exclude" prefix in the module config), to avoid it appearing as an insertable tag in the HannaCodeDialog dropdown menu.
      Example for an existing attribute named "image":
      image__options=[[_images_on_page]] And the code for the _images_on_page tag:
      <?php $image_names = array(); $image_fields = $page->fields->find('type=FieldtypeImage')->explode('name'); foreach($image_fields as $image_field) { $image_names = array_unique( array_merge($image_names, $page->$image_field->explode('name') ) ); } echo implode('|', $image_names); Choice of inputfield for attribute
      You can choose the inputfield that is used for an attribute in the dialog.
      For text attributes the supported inputfields are text (this is the default inputfield for text attributes so it isn't necessary to specify it if you want it) and textarea. Note: any manual line breaks inside a textarea are removed because these will break the CKEditor tag widget.
      Inputfields that support the selection of a single option are select (this is the default inputfield for attributes with options so it isn't necessary to specify it if you want it) and radios.
      Inputfields that support the selection of multiple options are selectmultiple, asmselect and checkboxes.
      You can also specify a checkbox inputfield - this is not for attributes with defined options but will limit an attribute to an integer value of 1 or 0.
      The names of the inputfield types are case-insensitive.
      Example for an existing attribute named "vegetables":
      vegetables__type=asmselect Descriptions and notes for inputfields
      You can add a description or notes to an attribute and these will be displayed in the dialog.
      Example for an existing attribute named "vegetables":
      vegetables__description=Please select vegetables for your soup. vegetables__notes=Pumpkin and celery is a delicious combination. Notes
      When creating or editing a Hanna tag you can view a basic cheatsheet outlining the HannaCodeDialog features relating to attributes below the "Attributes" config inputfield.
      Advanced
      Define or manipulate options in a hook
      You can hook HannaCodeDialog::prepareOptions to define or manipulate options for a Hanna tag attribute. Your Hanna tag must include a someattribute__options attribute in order for the hook to fire. The prepareOptions method receives the following arguments that can be used in your hook:
      options_string Any existing string of options you have set for the attribute attribute_name The name of the attribute the options are for tag_name The name of the Hanna tag page The page being edited If you hook after HannaCodeDialog::prepareOptions then your hook should set $event->return to an array of option values, or an associative array in the form of $value => $label.
      Build entire dialog form in a hook
      You can hook after HannaCodeDialog::buildForm to add inputfields to the dialog form. You can define options for the inputfields when you add them. Using a hook like this can be useful if you prefer to configure inputfield type/options/descriptions/notes in your IDE rather than as extra attributes in the Hanna tag settings. It's also useful if you want to use inputfield settings such as showIf.
      When you add the inputfields you must set both the name and the id of the inputfield to match the attribute name.
      You only need to set an inputfield value in the hook if you want to force the value - otherwise the current values from the tag are automatically applied.
      To use this hook you only have to define the essential attributes (the "fields" for the tag) in the Hanna Code settings and then all the other inputfield settings can be set in the hook.
      Example buildForm() hook
      The Hanna Code attributes defined for tag "meal" (a default value is defined for "vegetables"):
      vegetables=Carrot meat cooking_style comments The hook code in /site/ready.php:
      $wire->addHookAfter('HannaCodeDialog::buildForm', function(HookEvent $event) { // The Hanna tag that is being opened in the dialog $tag_name = $event->arguments(0); // Other arguments if you need them /* @var Page $edited_page */ $edited_page = $event->arguments(1); // The page open in Page Edit $current_attributes = $event->arguments(2); // The current attribute values $default_attributes = $event->arguments(3); // The default attribute values // The form rendered in the dialog /* @var InputfieldForm $form */ $form = $event->return; if($tag_name === 'meal') { $modules = $event->wire('modules'); /* @var InputfieldCheckboxes $f */ $f = $modules->InputfieldCheckboxes; $f->name = 'vegetables'; // Set name to match attribute $f->id = 'vegetables'; // Set id to match attribute $f->label = 'Vegetables'; $f->description = 'Please select some vegetables.'; $f->notes = "If you don't eat your vegetables you can't have any pudding."; $f->addOptions(['Carrot', 'Cabbage', 'Celery'], false); $form->add($f); /* @var InputfieldRadios $f */ $f = $modules->InputfieldRadios; $f->name = 'meat'; $f->id = 'meat'; $f->label = 'Meat'; $f->addOptions(['Pork', 'Beef', 'Chicken', 'Lamb'], false); $form->add($f); /* @var InputfieldSelect $f */ $f = $modules->InputfieldSelect; $f->name = 'cooking_style'; $f->id = 'cooking_style'; $f->label = 'How would you like it cooked?'; $f->addOptions(['Fried', 'Boiled', 'Baked'], false); $form->add($f); /* @var InputfieldText $f */ $f = $modules->InputfieldText; $f->name = 'comments'; $f->id = 'comments'; $f->label = 'Comments for the chef'; $f->showIf = 'cooking_style=Fried'; $form->add($f); } }); Troubleshooting
      HannaCodeDialog includes and automatically loads the third-party CKEditor plugins Line Utilities and Widget. If you have added these plugins to your CKEditor field already for some purpose and experience problems with HannaCodeDialog try deactivating those plugins from the CKEditor field settings.
    • By horst
      Wire Mail SMTP

      An extension to the (new) WireMail base class that uses SMTP-transport

      This module integrates EmailMessage, SMTP and SASL php-libraries from Manuel Lemos into ProcessWire. I use this continously evolved libraries for about 10 years now and there was never a reason or occasion not to do so. I use it nearly every day in my office for automated composing and sending personalized messages with attachments, requests for Disposition Notifications, etc. Also I have used it for sending personalized Bulkmails many times.

      The WireMailSmtp module extends the new email-related WireMail base class introduced in ProcessWire 2.4.1 (while this writing, the dev-branch only).
       
      Here are Ryans announcement.



      Current Version 0.4.2
      Changelog: https://github.com/horst-n/WireMailSmtp/blob/master/CHANGELOG.md
      get it from the Modules Directory Install and Configure

      Download the module into your site/modules/ directory and install it.

      In the config page you fill in settings for the SMTP server and optionaly the (default) sender, like email address, name and signature.
      You can test the smtp settings directly there. If it says "SUCCESS! SMTP settings appear to work correctly." you are ready to start using it in templates, modules or bootstrap scripts.


      Usage Examples
      The simplest way to use it:
      $numSent = wireMail($to, $from, $subject, $textBody); $numSent = wireMail($to, '', $subject, $textBody); // or with a default sender emailaddress on config page This will send a plain text message to each recipient.
       
      You may also use the object oriented style:
      $mail = wireMail(); // calling an empty wireMail() returns a wireMail object $mail->to($toEmail, $toName); $mail->from = $yourEmailaddress; // if you don't have set a default sender in config // or if you want to override that $mail->subject($subject); $mail->body($textBody); $numSent = $mail->send(); Or chained, like everywhere in ProcessWire:
      $mail = wireMail(); $numSent = $mail->to($toEmail)->subject($subject)->body($textBody)->send(); Additionaly to the basics there are more options available with WireMailSmtp. The main difference compared to the WireMail BaseClass is the sendSingle option. With it you can set only one To-Recipient but additional CC-Recipients.
      $mail = wireMail(); $mail->sendSingle(true)->to($toEmail, $toName)->cc(array('person1@example.com', 'person2@example.com', 'person3@example.com')); $numSent = $mail->subject($subject)->body($textBody)->send(); The same as function call with options array:
      $options = array( 'sendSingle' => true, 'cc' => array('person1@example.com', 'person2@example.com', 'person3@example.com') ); $numSent = wireMail($to, '', $subject, $textBody, $options); There are methods to your disposal to check if you have the right WireMail-Class and if the SMTP-settings are working:
      $mail = wireMail(); if($mail->className != 'WireMailSmtp') { // Uups, wrong WireMail-Class: do something to inform the user and quit echo "<p>Couldn't get the right WireMail-Module (WireMailSmtp). found: {$mail->className}</p>"; return; } if(!$mail->testConnection()) { // Connection not working: echo "<p>Couldn't connect to the SMTP server. Please check the {$mail->className} modules config settings!</p>"; return; }  
      A MORE ADVANCED DEBUG METHOD!
      You can add some debug code into a template file and call a page with it:
      $to = array('me@example.com'); $subject = 'Wiremail-SMTP Test ' . date('H:i:s') . ' äöü ÄÖÜ ß'; $mail = wireMail(); if($mail->className != 'WireMailSmtp') { echo "<p>Couldn't get the right WireMail-Module (WireMailSmtp). found: {$mail->className}</p>"; } else { $mail->from = '--INSERT YOUR SENDER ADDRESS HERE --'; // <--- !!!! $mail->to($to); $mail->subject($subject); $mail->sendSingle(true); $mail->body("Titel\n\ntext text TEXT text text\n"); $mail->bodyHTML("<h1>Titel</h1><p>text text <strong>TEXT</strong> text text</p>"); $dump = $mail->debugSend(1); } So, in short, instead of using $mail->send(), use $mail->debugSend(1) to get output on a frontend testpage.
      The output is PRE formatted and contains the areas: SETTINGS, RESULT, ERRORS and a complete debuglog of the server connection, like this one:
       
      Following are a ...


      List of all options and features


      testConnection () - returns true on success, false on failures


      sendSingle ( true | false ) - default is false

      sendBulk ( true | false ) - default is false, Set this to true if you have lots of recipients (50+)


      to ($recipients) - one emailaddress or array with multiple emailaddresses

      cc ($recipients) - only available with mode sendSingle, one emailaddress or array with multiple emailaddresses

      bcc ($recipients) - one emailaddress or array with multiple emailaddresses

       
      from = 'person@example.com' - emailaddress, can be set in module config (called Sender Emailaddress) but it can be overwritten here

      fromName = 'Name Surname' - optional, can be set in module config (called Sender Name) but it can be overwritten here


      priority (3) - 1 = Highest | 2 = High | 3 = Normal | 4 = Low | 5 = Lowest

      dispositionNotification () or notification () - request a Disposition Notification


      subject ($subject) - subject of the message

      body ($textBody) - use this one alone to create and send plainText emailmessages

      bodyHTML ($htmlBody) - use this to create a Multipart Alternative Emailmessage (containing a HTML-Part and a Plaintext-Part as fallback)

      addSignature ( true | false ) - the default-behave is selectable in config screen, this can be overridden here
      (only available if a signature is defined in the config screen)

      attachment ($filename, $alternativeBasename = "") - add attachment file, optionally alternative basename


      send () - send the message(s) and return number of successful sent messages


      debugSend(1) - returns and / or outputs a (pre formatted) dump that contains the areas: SETTINGS, RESULT, ERRORS and a complete debuglog of the server connection. (See above the example code under ADVANCED DEBUG METHOD for further instructions!)


      getResult () - returns a dump (array) with all recipients (to, cc, bcc) and settings you have selected with the message, the message subject and body, and lists of successfull addresses and failed addresses,


      logActivity ($logmessage) - you may log success if you want

      logError ($logmessage) - you may log warnings, too. - Errors are logged automaticaly
       
       
      useSentLog (true | false) - intended for usage with e.g. third party newsletter modules - tells the send() method to make usage of the sentLog-methods - the following three sentLog methods are hookable, e.g. if you don't want log into files you may provide your own storage, or add additional functionality here

      sentLogReset ()  - starts a new LogSession - Best usage would be interactively once when setting up a new Newsletter

      sentLogGet ()  - is called automaticly within the send() method - returns an array containing all previously used emailaddresses

      sentLogAdd ($emailaddress)  - is called automaticly within the send() method
      Changelog: https://github.com/horst-n/WireMailSmtp/blob/master/CHANGELOG.md
       
       
    • By Robin S
      Sanitizer EasySlugger
      Allows the use of the EasySlugger library as Sanitizer methods.
      Installation
      Install the Sanitizer EasySlugger module.
      Usage
      The module adds four new sanitizer methods.
      slugger($string, $options)
      Similar to $sanitizer->pageName() - I'm not sure if there are any advantages over that method. Included because it is one of the methods offered by EasySlugger.
      $slug = $sanitizer->slugger('Lorem Ipsum'); // Result: lorem-ipsum utf8Slugger($string, $options)
      Creates slugs from non-latin alphabets.
      $slug = $sanitizer->utf8Slugger('这个用汉语怎么说'); // Result: zhe-ge-yong-han-yu-zen-me-shuo seoSlugger($string, $options)
      Augments the string before turning it into a slug. The conversions are related to numbers, currencies, email addresses and other common symbols.
      $slug = $sanitizer->seoSlugger('The price is $5.99'); // Result: the-price-is-5-dollars-99-cents See the EasySlugger readme for some more examples.
      seoUtf8Slugger($string, $options)
      A combination of utf8Slugger() and seoSlugger().
      $slug = $sanitizer->seoUtf8Slugger('价钱是 $5.99'); // Result: jia-qian-shi-5-dollars-99-cents $options argument
      Each of the methods can take an $options array as a second argument.
      separator (string): the character that separates words in the slug. Default: - unique (bool): Determines whether a random suffix is added at the end of the slug. Default: false $slug = $sanitizer->utf8Slugger('这个用汉语怎么说', ['separator' => '_', 'unique' => true]); // Result: zhe_ge_yong_han_yu_zen_me_shuo_3ad66c4  
      https://github.com/Toutouwai/SanitizerEasySlugger
      https://modules.processwire.com/modules/sanitizer-easy-slugger/
    • By Robin S
      If you've ever needed to insert links to a large number of files within CKEditor you may have found that the standard PW link modal is a somewhat slow way to do it.
      This module provides a quicker way to insert links to files on the page being edited. You can insert a link to an individual file, or insert an unordered list of links to all files on the page with a single click.
      CKEditor Link Files
      Adds a menu to CKEditor to allow the quick insertion of links to files on the page being edited.

      Features
      Hover a menu item to see the "Description" of the corresponding file (if present). Click a menu item to insert a link to the corresponding file at the current cursor position. The filename is used as the link text. If you Alt-click a menu item the file description is used as the link text (with fallback to filename if no description entered). If text is currently selected in the editor then the selected text is used as the link text. Click "* Insert links to all files *" to insert an unordered list of links to all files on the page. Also works with the Alt-click option. Menu is built via AJAX so newly uploaded files are included in the menu without the page needing to be saved. However, descriptions are not available for newly uploaded files until the page is saved. There is an option in the module config to include files from Repeater fields in the edited page. Nested Repeater fields (files inside a Repeater inside another Repeater) are not supported. Installation
      Install the CKEditor Link Files module.
      For any CKEditor field where you want the "Insert link to file" dropdown menu to appear in the CKEditor toolbar, visit the field settings and add "LinkFilesMenu" to the "CKEditor Toolbar" settings field.
       
      http://modules.processwire.com/modules/cke-link-files/
      https://github.com/Toutouwai/CkeLinkFiles
    • By David Karich
      ProcessWire InputfieldRepeaterMatrixDuplicate
      Thanks to the great ProModule "RepeaterMatrix" I have the possibility to create complex repeater items. With it I have created a quite powerful page builder. Many different content modules, with many more possible design options. The RepeaterMatrix module supports the cloning of items, but only within the same page. Now I often have the case that very design-intensive pages and items are created. If you want to use a content module on a different page (e.g. in the same design), you have to rebuild each item manually every time.
      This module extends the commercial ProModule "RepeaterMatrix" by the function to duplicate repeater items from one page to another page. The condition is that the target field is the same matrix field from which the item is duplicated. This module is currently understood as proof of concept. There are a few limitations that need to be considered. The intention of the module is that this functionality is integrated into the core of RepeaterMatrix and does not require an extra module.
      Check out the screencast
      What the module can do
      Duplicate a repeater item from one page to another No matter how complex the item is Full support for file and image fields Multilingual support Support of Min and Max settings Live synchronization of clipboard between multiple browser tabs. Copy an item and simply switch the browser tab to the target page and you will immediately see the past button Support of multiple RepeaterMatrix fields on one page Configurable which roles and fields are excluded Duplicated items are automatically pasted to the end of the target field and set to hidden status so that changes are not directly published Automatic clipboard update when other items are picked Automatically removes old clipboard data if it is not pasted within 6 hours Delete clipboard itself by clicking the selected item again Benefit: unbelievably fast workflow and content replication What the module can't do
      Before an item can be duplicated in its current version, the source page must be saved. This means that if you make changes to an item and copy this, the old saved state will be duplicated Dynamic loading is currently not possible. Means no AJAX. When pasting, the target page is saved completely No support for nested repeater items. Currently only first level items can be duplicated. Means a repeater field in a repeater field cannot be duplicated. Workaround: simply duplicate the parent item Dynamic reloading and adding of repeater items cannot be registered. Several interfaces and events from the core are missing. The initialization occurs only once after the page load event Changelog
      1.0.4
      Bug fix: Various bug fixes and improvements in live synchronization Bug fix: Items are no longer inserted when the normal save button is clicked. Only when the past button is explicitly clicked Feature: Support of multiple repeater fields in one page Feature: Support of repeater Min/Max settings Feature: Configurable roles and fields Enhancement: Improved clipboard management Enhancement: Documentation improvement Enhancement: Corrected few typos #1 1.0.3
      Feature: Live synchronization Enhancement: Load the module only in the backend Enhancement: Documentation improvement 1.0.2
      Bug fix: Various bug fixes and improvements in JS functions Enhancement: Documentation improvement Enhancement: Corrected few typos 1.0.1
      Bug fix: Various bug fixes and improvements in the duplication process 1.0.0
      Initial release Support this module
      If this module is useful for you, I am very thankful for your small donation: Donate 5,- Euro (via PayPal – or an amount of your choice. Thank you!)
      Download this module
      > Github: https://github.com/FlipZoomMedia/InputfieldRepeaterMatrixDuplicate
      > PW module directory: https://modules.processwire.com/modules/inputfield-repeater-matrix-duplicate/
×
×
  • Create New...