Jump to content
horst

Wire Queue, basic implementation of simple queues

Recommended Posts

Wire Queue

Wire Queue is a module that allows easy creation and usage of Queues in ProcessWire.

It is based upon a basic parent module (WireQueue) that should have one or multiple StorageHandler modules installed too. This beta release contains a simple plain text storage module, WireQueueTextfile, and a simple Sqlite3-DB storage module, WireQueueSqlite3.

The base module creates the needed:

  • FIELDS (wire_queue_type, wire_queue_state, wire_queue_storage_options)
  • TEMPLATES (wire-queue-container, wire-queue-storage, wire-queue-tools)
  • PAGES (wire-queues = container for all Queuepages, wire-queue-storages = container for StoragetypePages)
  • ROLE (wire-queue-admin)

Each storage module creates one page under wire-queue-storages.

New Queues can be created in the backend by adding a new page under "Wire Queues". After creation one have to select a Storage type from availables list and publish the page. After that, there are some buttons available to start / pause / and close the queue.

Putting and getting data to and from the queue is done via API calls. First you have to get the page that holds the queue object.

// get and validate the queue handle
if($queue = $pages->get(SELECTOR_TO_DESIRED_PAGE)->wireQueue()) {
    // wrap your data into an array and pass it into the queue
    $success = $queue->addItem($data);
    ...
}
// get and validate the queue handle
if($queue = $pages->get(SELECTOR_TO_DESIRED_PAGE)->wireQueue()) {
    $data = $queue->getItem();
    ...
}

Thats basically all what you want from a queue. Following are a few conveniences:

  • $queue->getPage()->title gives you the title of the queue, ($queue->getPage() returns the page object)
  • $queue->className() shows the StorageType of the queue
  • $queue->getState() and $queue->getStateStr() returns the current state of a queue:
    1 = new / empty
    2 = enabled / running
    3 = paused
    4 = closed / archived
  • $queue->itemCount() gives the total number of all pending items in the queue

Here is code that gives an overview of all Queues in the system:

$container = $pages->get('template=wire-queue-container');
$bodyContent = "<h1>$container->title</h1>";
$bodyContent .= "<p>There are currently {$container->numChildren} Queues defined:</p>";
$bodyContent .= "<ul>";
foreach($container->children() as $p) {
    if(! ($queue = $p->wireQueue())) continue;
    $bodyContent .= "<li>{$queue->getPage()->title}<ul>";
    if(!$queue->ready2use()) {
        $bodyContent .= "<li>{$queue->className}</li>";
        $bodyContent .= "<li>This Storagetype is not ready to use! The System seems not to provide all requirements.</li>";
        $bodyContent .= "</ul></li>";
        continue;
    }
    $bodyContent .= "<li>{$queue->className}</li>";
    $bodyContent .= "<li>{$queue->getStateStr()} ({$queue->getState()})</li>";
    $bodyContent .= "<li>Currently are {$queue->itemCount()} items pending!</li>";
    $bodyContent .= "</ul></li>";
}
$bodyContent .= "</ul>";

Following is a screenshot of the backend

wire-queue-01.gif

The module is available in the modules directory: http://modules.processwire.com/modules/wire-queue/
Or you get it from Github: https://github.com/horst-n/WireQueue.
.
.
.
The Sqlite3 storage handler not only let you push and pull data items to and from it, it also can keep track of the current state of a record. If you use multiple / different workers for pulling and processing the data, you can store an ID for them too.
 
This is how the DB Table looks like:
pw_wirequeue_sqlite3_db-screen.jpg
 
The Wire Queue Sqlite3 storage handler provides the methods

  • addItem($arrayData)      // same as WireQueueTextfile
  • getItem($worker = null)   // same as WireQueueTextfile, (but the textfile storage cannot support $worker!)
  • updateItemState($id, $state, $worker = null)  // this you can use for further processing, if you want

 .
addItem($arrayData)
 
$arrayData is a, maybe associative, array containing all your data for one record. The method stores it under the next free id under data in the Sqlite-DB-file and sets the state to 0. The field worker is empty for new added records.

Following is a basic working example for pushing data records into a queue:

// you have created a queue in PW, the ID of the page is 1420 for example
// here is some skeleton code for an importer that uses this queue

// get and validate the queue handle
if(! ($queue = $pages->get('id=1420')->wireQueue())) exit(); // we could not get the Queuepage

// now start to scan / read data for your imports, wrap each record into an array and put it into the queue
foreach($pages->find(YOURSELECTOR) as $p) {
    $data = array($p->field1, $p->field2);
    $queue->addItem($data);
}

.

.

getItem($worker = null)
 
$worker is an integer that you can define to suite your needs. If you don't use or don't want identify multiple workers, just ommit it. The method pulls one pending record from the queue, changes the state from 0 to 1, and returns an associative array with the keys id and data. array('id' => integer, 'data' => array)
You will need the id if you further want to use the queue to keep track of processing steps.
You must pull your stored $data from $array['data'] and use the id for further storing the state.

.

updateItemState($id, $state, $worker = null)
 
$id identifys the record a worker has processed, for $state you can define and use every integer you like, but not 0 or 1. If you also want to store altered data, and not only different states, you can use updateItem($id, $state, $worker = null, $data = null) instead of updateItemState().

.

Here is a working example with a bit pseudo code how workers can get a pending record to process it further and store back the result of the process:

// you have created a queue in PW, the ID of the page is 1420 for example
// here is some skeleton code for an importer that uses this queue

// get and validate the queue handle
if(! ($queue = $pages->get('id=1420')->wireQueue())) exit(); // we could not get the Queuepage

// we do not use different workers here in that example, so we do not pass an worker id here
$tmp = $queue->getItem();   // get a record from the queue
$record_id = $tmp['id'];    // store the record id
$data = $tmp['data'];       // get the data array

// process the $data ... 
// and get the result of the process, in this example true or false
$result = processMyRecord($data);

// as new records have a state = 0, fetched records have a state = 1, 
// you may define a state of 2 for successful processed records, and higher ones for failed records
$state = true === $result ? 2 : 3;
$queue->updateItemState($record_id, $state);

.

.

getItem4FurtherProcessing($state, $worker = null)
 
The $state you pass to the method is the state you want get the record for. If there is one pending, its state will be set +1 and the id and data is passed back to you in an associative array: array('id' => integer, 'data' => array).

---------

Here is a pseudo code example how (multiple) worker scripts may batch process queue records with the sqlite storage handler

// on the server in this example, everyscript will timeout / die after 180 seconds
// we start a timer
$time = Debug::timer();

// we use different instances of workers that pull and process records from the queue, 
// so additionally to the processings states, we also want to store the worker ids
$worker = 2000;

// now start to process items by pulling one after the other from the queue
while(150 > Debug::timer($time)) {
    $tmp = $queue->getItem($worker);   // get a record from the queue
    if(!$tmp) continue;                // maybe currently are no pending records available
    $record_id = $tmp['id'];           // store the record id
    $data = $tmp['data'];              // get the data array
    $result = processMyRecord($data);  // process the data and get back the result
    $state = true === $result ? 2 : 3; // define an integer for the new state, according to the result of processing the data
    $queue->updateItemState($record_id, $state, $worker);
}

// we are close to the timeout for the script, so finish it with calling it itself to get a fresh run
$session->redirect('./');
  • Like 10

Share this post


Link to post
Share on other sites

Just to clarify: the further processing does not have to be done from within Processwire, nor does it to be done by PHP. You can use PW to only collect the data, maybe in the example Sqlite storage handler. To further process the data records, you can use every software that can communicates to a Sqlite DBfile.

If you need different fields in a Sqlite table, just copy the WireQueueSqlite3 module, - change the file- and class name to be no duplicates of the distributed storage handler, - and, at least, change the fielddefinitions for the DB table. But you can change every thing you want.

If you want or need to use other storage types and setup a new WireQueueSomething module, - please feel free to submit / commit it. :)

Share this post


Link to post
Share on other sites

hi horst,

thank you for your module. could be that i need it one day, though i have to say it was all new to me and your explanation in the other thread was helpful for me: https://processwire.com/talk/topic/11927-wirequeue-needs-testing/?p=111287 :)

thank you also for the screencasts that helped me a lot understanding what it is all about.

the further processing must not be done from within Processwire...

i think you mean "does not have to be done" as "must not be done" = "darf nicht innerhalb pw gemacht werden", right?

could you please explain why you used TXT/SQLite for data storage? why not use a mysql table from within pw? thank you :)

Share this post


Link to post
Share on other sites

@BernhardB: thanks for the language explanation, I have corrected it above in my post! :)

I haven't used MySQL tables, because my main motivation was to have a storage that not belongs to the PW-DB. I think this is useful in regard of Backup / Restore processes, if one have not mass temporary records with it. Filebased storage is portable and easy to use (TXT). SQLite is also portable but has DB support what lets you simply push, pull and update records. For example, think of sending newsletters to 5k recipients, SQLite is very convenient to handle this.

Another usecase what I have had: A website collected infos over a time of two weeks and the (post) processing should be done afterwards. Here we simply have implemented a download link for the customer into the editpage, so that he could store the textfile (CSV style), locally on his computer for further processing with excel.

So, I believe there will be also usecases where a MySQL table is a better storage. For all storage types that will need initial config params, I have implemented a simple Textareafield into the queue page, what can be used to take that settings. (one can use it as key = value pairs each in its own row, like in some other modules, maybe. But it is up to the developer if he like to use that simple approach or if he implements something other). The Textareafield is collapsed by default for admins, and hidden for others. (Whereas the queue pages require minimum the role "wire-queue-admin" or superuser for everything else than viewing the state or count).

So, hopefully someone will make a mysql storage handler and commit it here or at github. :)

(or another DB)

Edited by horst
  • Like 1

Share this post


Link to post
Share on other sites

Hey Horst, 

Looks really impressive, but I'm wondering how this module differs from the queuing done in IftRunner combined with Page Actions. I'm comparing both at the moment for a rather large project we are working on. One thing I see it that you have different queues whereas IftRunner only has one queue to store them all. Any other differences that you are aware of?

Share this post


Link to post
Share on other sites

Hi Arjen,

I haven't used IftRunner and doesn't know it. So, sorry, I cannot answer anything in regard of comparing. If you already know IftRunner and it does support what you need, I would stick with it. WireQueue is new and not (widely) tested currently. In fact, currently it is only tested by me in a few use cases. Before using it in a huge project, you definetly should do thouroughful testing.

With a quick look through the repository, it seems that queuing in IftRunner is done internally, means, it is mapped to page actions, you do not have to populate it manually, but you also cannot use it externally, without page actions. But not really sure on this, - it was my first read here.

Share this post


Link to post
Share on other sites

Thanks for responding. We are going to test it thoroughly. That's why I'm looking for all options. Pity you haven't used it yet. IftRunner with (Page) Actions are really simple as a concept, but quite powerful. Antti wrote about their use case a while ago.

  • Like 2

Share this post


Link to post
Share on other sites

...  Pity you haven't used it yet. IftRunner with (Page) Actions are really simple as a concept, but quite powerful. Antti wrote about their use case a while ago.

Thanks for the link to Anttis writeup.

------------------

Following is a description for what the SQLite handler was used in the past before I built WireQueue around it.

One use case was for an award winning Cinemaportal where they send up to 2x 13k emails in different newsletters every week. I should embed the WireMailSMTP module there and I suggested that we should use SQLite files for it.

This is running a half year in production now without any problems. At the beginning we have had a testing for 2 weeks where we slightly modified and tweeked some little parts. On that server where the website is hosted were running many many different programs and services, so, that for unknown reasons, sometimes the SQLite DB stopped working unexpected. But SQLite has a sort of transaction feature (Sqlite-Journal files), so that no recipient got lost. On every run we retrieve 500 pending recipients.

The only thing what could happen there was, that, when immediately after successful sending a mail and the SQLite crashed before it could update the record in the dbfile AND in the journal file, this single recipient will be picked up again with the next batch, what results in sending him the newsletter twice. So, this (only) happened a few times per week, (3-10) on 10k mails, = 0,1 percent. And as this may be a bit specific to that explicit server, as I haven't encountered it with other use cases, it seems to be much more secure to use SQLite with its transaction / rollback functionality as MySQL with MyISAM! Don't know about the differences when using MySQL and InnoDB. But we / you can build storage handlers for every DB we / you like. :)

So, what I have done with building WireQueue, is implementing it for easy usage and monitoring in PWs backend. post-1041-0-54734100-1455109217_thumb.pn

The previous used fieldsdefinition of the SQLite table has changed from userid, email, sent, tries to data, state, worker, but everything else of the SQLite part is identical in WireQueue.

Hope this helps a bit.

Edited by horst
added info about use cases for SQLite
  • Like 3

Share this post


Link to post
Share on other sites

hi horst,

doing first tests with your module: how can i re-open a queue after closing/archiving it?

thank you

Share this post


Link to post
Share on other sites

With the UI, you cannot! Archiveing means what it says. :)

If you want to stop using it, with the possibility to reuse it later, please use pause.

I'm not sure on this, as it isn't meant to do this by design, but you may try to cahnge the fieldvalue by API:

// get the desired queue page by its id
$queue = $pages->get("id=1234");

// get the fieldname that stores the state
$fieldname = WireQueue::WIRE_QUEUE_FIELD2;

// change value from 4 (archived) to 3 (paused), or 2 (active)
$queue->$fieldname = 3; // or 2
$queue->save();

BUT: Without warranty. I'm not sure if this has any sideeffects, its not tested!

Share this post


Link to post
Share on other sites
$pages->get(SELECTOR_TO_DESIRED_PAGE)->WireQueue() 

You have typos in your first two examples: it should be wireQueue and not WireQueue

ok no problem, but i think there should at least be a warning that this is an irreversible action :)

Share this post


Link to post
Share on other sites
$pages->get(SELECTOR_TO_DESIRED_PAGE)->WireQueue() 

You have typos in your first two examples: it should be wireQueue and not WireQueue

ok no problem, but i think there should at least be a warning that this is an irreversible action :)

Thanks for pointing to the typos. I corrected it.

  • Like 1

Share this post


Link to post
Share on other sites

hi horst,

i need a little more help, sorry. i don't really get the concept. for example when i have a queue to resize images, how would i setup this? when i have a queue "resizeimages" i could add all resize requests to this queue and then run the resize for every item. so far so good, but what if i have different users with images from different pages? user A adds 100 images to queue "resizeimages" and user B adds 100 images to the same queue. the queue now has 200 images in it if that happens at the same time. how does the queue know which images belong to wich user?

the "worker" is empty for new entries so i can't use this. or should i do an additem() and right after that an updateitemstate(...worker...)?

or am i completely on the wrong track with what i want to do?

thank you

edit: ok i don't think that would work because there would still be all items in the queue and there's no kind of "selector" to get only the items from one worker, right? so would i have to create a queue for every page where i want to resize images? i didn't try it so far, but i think that sould be possible via standard api creating new pages and setting the appropriate values?

Share this post


Link to post
Share on other sites

Sorry, I cannot follow. A queue only stores the data you put into it, nothing more. If you need references to whatever, put it into your data. That's the only way to have your needed data together. But this has nothing to do with the queue. Once again, the queue is nothing else than a storage, a DB table, with exactly 1 field for your data. Therefore, (I think you do this already), you have to wrap your data into an array. If you need more data-information as you currently have in it, just put it into the data array too. :)

Share this post


Link to post
Share on other sites

hmmm, ok i see you really didn't get me :) i was writing a more detailed answer but while i was writing i found a better solution for my usecase. i think i wanted to use the queue in a way it is not built for (but not the way you understood me, but thats no factor any more) ;)

Share this post


Link to post
Share on other sites

hi horst, me again ;)

just came across https://processwire.com/api/variables/log/ and as i read the docs i thought it is quite similar to txt-queues. i think one could easily create simple queues this way and would be interested in your opinion.

creating a new queue would be as simple as $log->save('examplequeue', 'mydata');

fetching lines from the end of the file is also easy (http://kongondo.github.io/ProcessWireAPIGen/master/source-class-WireLog.html#196-229) and could maybe modified by ryan with an option go get lines from the beginning of the file and not only the end. this functionality is already built into the filelog class http://kongondo.github.io/ProcessWireAPIGen/master/source-class-FileLog.html#164-216

only thing missing would be a method to delete rows from the beginning or end of the file, but that would also be really easy.

edit: don't know about the "workers", didn't get the concept...

Share this post


Link to post
Share on other sites

edit: don't know about the "workers", didn't get the concept...

Workers are separate processes, which run alongside your webserver stack on the host system. They can process things they get out of the queue without being reliant on the Request->Process->Response cycle of the webserver.

E.g. You want to have a way for users to request a big report on your website. As a big report sounds like, it take some time to generate the pdf for it (say ~1 Min). You wouldn't want your user to sit around and watch the nice browser spinner, waiting for the next page to pop up. You can now write the request in a queue and instantly return a message stating that the report will be generated and emailed to the user when ready.

To generate that report you may know the option of using a cron to trigger website logic, without an actual user requesting a website, but that's also flawed with things like timeouts and such, also it's only starting in intervals. That's where workers come in.

These are php (and other script) processes, which often run as services like e.g. your apache server does. These scripts contain some way to not exit early (e.g. infinite while loop) – which is the biggest difference to website request – and periodically check their queue if there's work to do. If your report request is in it, they generate the pdf, email it and if done delete the queue item. If there are other reports to do they'll move on to them, otherwise they sleep until another queue item is available. 

  • Like 5

Share this post


Link to post
Share on other sites

@horst

Thanks for this module, I've been hammering it pretty hard for the past month and have just created a pull-request over on github for your consideration. I've been having a few reliability problems with the textfile driver's itemCount() method. I also wanted a fast way to purge a queue without having to call getItem() repeatedly.

The itemCount() method sometimes returns an incorrect count (anywhere from 0 to the true number of items in the queue). This happens even when there are multiple thousands of jobs queued up and it makes it unreliable as part of a conditional statement. The explanation is given in the commit that fixes the issue as part of the pull request.

Thanks for looking!

  • Like 6

Share this post


Link to post
Share on other sites

Many thanks @netcarver.

I must admit that I have written the Textfile-driver only for this module to have a basic driver. Other as the SQLite-driver, it has not have much testing. I'm glad to hear that you have done this and you contribute to the module. I have you added as colaborator to it, so that you can change / add other improvements / enhancements as you like. ^-^;)

  • Like 3

Share this post


Link to post
Share on other sites

Thanks for adding me as a colaborator, @horst.

I've pushed a new dev branch to the repository that contains,

  • Bugfixes to the textfile driver's itemCount() method and small cleanups for edge cases.
  • New methods isEmpty(), purgeItems() and getItems() added to the base storage class and fully implemented in the textfile driver. The SQLite driver currently has stub implementations of these that need filling out.
  • The addition of a Redis driver for super-fast in-memory queues (implements the full base class interface.)

If anyone wants to give this a try before horst gets around to merging into the master branch, feel free to grab the zip (repository here.)

  • Like 5

Share this post


Link to post
Share on other sites

Horst and I would like to announce the release of v1.0.0 of the WireQueue suite of modules.

There are now drivers for textfiles, redis and SQLite and the interface has been extended to include the isEmpty(), purgeItems() and getItems() methods.

Enjoy!

  • Like 12

Share this post


Link to post
Share on other sites

I think I may have found a small problem with WireQueueTextfile, although it could be due to this PHP bug?

The array is being serialized in the addItem() function, but in my case an element's value is sometimes a string containing newline characters "\r\n". I believe WireQueueTextfile saves each array to one line, but these characters are causing the serialized string to be split onto multiple lines, which I'm assuming causes problems for WireQueue.

Would it make sense to somehow sanitize these values (outside of WireQueue) before adding them to the queue?

Share this post


Link to post
Share on other sites

Could you try replacing addItem() and getItem() with the following...

    public function addItem($arrayData) {
        if(!$this->_addItem()) return false;
        if(2 != $this->getState()) return false;
        if(!$fp = @fopen($this->getFilename(), 'ab')) return false;
        if(flock($fp, LOCK_EX)) {
            $arrayData = serialize($arrayData);
            $data = str_replace("\r\n", '\r\n', $arrayData) . "\n";
            $res = fwrite($fp, $data);
            fflush($fp);
            flock($fp, LOCK_UN);
            fclose($fp);
            return $res == strlen($data);
        }
        fclose($fp);
        return false;
    }

    public function getItem($worker = null) {
        if(!$this->_getItem()) return false;
        if(2 != $this->getState()) return false;
        if(!$fp = @fopen($this->getFilename(), 'rb+')) return false;
        if(flock($fp, LOCK_EX)) {
            $line = trim(fgets($fp));
            if(!$line) {
                flock($fp, LOCK_UN);
                fclose($fp);
                if(0 == $this->itemCount()) return null;
                return false;
            }
            // we have the first entry, now write all following data into a buffer
            $fpTmp = @fopen('php://temp/maxmemory:' . intval(1024 * 1024 * 5), 'rb+');
            while(!feof($fp)) fwrite($fpTmp, fread($fp, 4096));
            fseek($fp, 0, SEEK_SET);
            ftruncate($fp, 0);
            fseek($fpTmp, 0, SEEK_SET);
            // write back buffer into file
            while(!feof($fpTmp)) fwrite($fp, fread($fpTmp, 4096));
            fclose($fpTmp);
            fflush($fp);
            flock($fp, LOCK_UN);
            fclose($fp);
        }
        return unserialize(str_replace('\r\n', "\r\n", $line));
    }

Untested - YMMV!

  • Like 2

Share this post


Link to post
Share on other sites

No luck with that unfortunately, still lots of fragmented lines. Do the instances of '\r\n'  need to be wrapped in double quotes? I can't work out the logic here!

I should also note that these strings can include \r\n but are also sometimes littered with instances of \r or \n individually.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By BitPoet
      Here's a little text formatter module that redirects all external links found in a textarea (e.g. for hit counting) to a given URL.
      TextformatterExternalRedirect
      GitHub repo
      As TextFormatters do, this module modifies the contents of the textarea (HTML) at rendering time. It prepends the given URL to the link address for external links (http and https), optionally makes sure that the link is opened in a new window and allows you to add css classes to such links.
      All options can be configured in the module's configuration settings.
      Usage:
      Download the zip archive and extract it to site/modules, the rename the top folder to TextformatterExternalRedirect Go to the backend and select "Modules" -> "Refresh", then click "Install" for "External Link Redirect" Click "Settings" and configure the module
      Go into the field configuration for the field(s) you want this formatter to apply to and add it from the dropdown
      Done, now any external links in the configured fields will have the configured settings applied Feel free to leave feedback and post and questions you have here.
       
    • By teppo
      Hey folks!
      Took a couple of late nights, but managed to turn this old gist of mine into a proper module. The name is SearchEngine, and currently it provides support for indexing page contents (into a hidden textarea field created automatically), and also includes a helper feature ("Finder") for querying said contents. No fancy features like stemming here yet, but something along those lines might be added later if it seems useful (and if I find a decent implementation to integrate).
      Though the API and selector engine make it really easy to create site search pages, I pretty much always end up duplicating the same features from site to site. Also – since it takes a bit of extra time – it's tempting to skip over some accessibility related things, and leave features like text highlighting out. Overall I think it makes sense to bundle all that into a module, which can then be reused over and over again 🙂
      Note: markup generation is not yet built into the module, which is why the examples below use PageArray::render() method to produce a simple list of results. This will be added later on, as a part of the same module or a separate Markup module. There's also no fancy JS API or anything like that (yet).
      This is an early release, so be kind – I got the find feature working last night (or perhaps this morning), and some final tweaks and updates were made just an hour ago 😅
      GitHub repository: https://github.com/teppokoivula/SearchEngine Modules directory: https://modules.processwire.com/modules/search-engine/ Demo: https://wireframe-framework.com/search/ Usage
      Install SearchEngine module. Note: the module will automatically create an index field install time, so be sure to define a custom field (via site config) before installation if you don't want it to be called "search_index". You can change the field name later as well, but you'll have to update the "index_field" option in site config or module settings (in Admin) after renaming it.
      Add the site search index field to templates you want to make searchable. Use selectors to query values in site search index. Note: you can use any operator for your selectors, you will likely find the '=' and '%=' operators most useful here. You can read more about selector operators from ProcessWire's documentation.
      Options
      By default the module will create a search index field called 'search_index' and store values from Page fields title, headline, summary, and body to said index field when a page is saved. You can modify this behaviour (field name and/or indexed page fields) either via the Module config screen in the PocessWire Admin, or by defining $config->SearchEngine array in your site config file or other applicable location:
      $config->SearchEngine = [ 'index_field' => 'search_index', 'indexed_fields' => [ 'title', 'headline', 'summary', 'body', ], 'prefixes' => [ 'link' => 'link:', ], 'find_args' => [ 'limit' => 25, 'sort' => 'sort', 'operator' => '%=', 'query_param' => null, 'selector_extra' => '', ], ]; You can access the search index field just like any other ProcessWire field with selectors:
      if ($q = $sanitizer->selectorValue($input->get->q)) { $results = $pages->find('search_index%=' . $query_string . ', limit=25'); echo $results->render(); echo $results->renderPager(); } Alternatively you can delegate the find operation to the SearchEngine module as well:
      $query = $modules->get('SearchEngine')->find($input->get->q); echo $query->resultsString; // alias for $query->results->render() echo $query->pager; // alias for $query->results->renderPager() Requirements
      ProcessWire >= 3.0.112 PHP >= 7.1.0 Note: later versions of the module may require Composer, or alternatively some additional features may require installing via Composer. This is still under consideration – so far there's nothing here that would really depend on it, but advanced features like stemming most likely would.
      Installing
      It's the usual thing: download or clone the SearchEngine directory into your /site/modules/ directory and install via Admin. Alternatively you can install SearchEngine with Composer by executing composer require teppokoivula/search-engine in your site directory.
    • By teppo
      MarkupMenu is a markup module for generating menu trees. When provided a root page as a starting point, it generates a navigation tree (by default as a HTML "<ul>" element wrapped by a "<nav>" element) from that point onwards. If you've also provided it with current (active) page, the menu will be rendered accordingly, with current item highlighted and items rendered up to that item and its children (unless you disable the "collapsed" option, in which case the full page tree will be rendered instead).
      Modules directory: https://modules.processwire.com/modules/markup-menu/ GitHub repository: https://github.com/teppokoivula/MarkupMenu Usage
      As a markup module, MarkupMenu is intended for front-end use, but you can of course use it in a module as well. Typically you'll only need the render() method, which takes an array of options as its only argument:
      echo $modules->get('MarkupMenu')->render([ 'root_page' => $pages->get(1), 'current_page' => $page, ]); Note: if you omit root_page, site root page is used by default. If you omit current_page, the menu will be rendered, but current (active) page won't be highlighted etc.
      A slightly more complex example, based on what I'm using on one of my own sites to render a (single-level) top menu:
      echo $modules->get('MarkupMenu')->render([ 'current_page' => $page, 'templates' => [ 'nav' => '<nav class="{classes} menu--{menu_class_modifier}" aria-label="{aria_label}">%s</nav>', 'item_current' => '<a class="menu__item menu__item--current" href="{item.url}" tabindex="0" aria-label="Current page: {item.title}">{item.title}</a>', ], 'placeholders' => [ 'menu_class_modifier' => 'top', 'aria_label' => 'Main navigation', ], 'include' => [ 'root_page' => true, ], 'exclude' => [ 'level_greater_than' => 1, ], ]); Note: some things you see above may not be entirely sensible, such as the use of {menu_class_modifier} and {aria_label} placeholders. On the actual site the "nav" template is defined in site config, so I can define just these parts on a case-by-case basis while actual nav markup is maintained in one place.
      Please check out the README file for available render options. I'd very much prefer not to keep this list up to date in multiple places. Basically there are settings for defining "templates" for different parts of the menu (list, item, etc.), include array for defining rules for including in the menu and exclude array for the opposite effect, classes and placeholders arrays for overriding default classes and injecting custom placeholders, etc. 🙂
      MarkupMenu vs. MarkupSimpleNavigation
      TL;DR: this is another take on the same concept. There are many similarities, but also some differences – especially when it comes to the supported options and syntax. If you're currently using MarkupSimpleNavigation then there's probably no reason to switch over.
      I'd be surprised if someone didn't draw lines between this module and Soma's awesome MarkupSimpleNavigation. Simply put I've been using MSN (...) for years, and it's been great – but there are some issues with it, particularly in the markup generation area, and it also does some things in a way that doesn't quite work for me – the xtemplates thing being one of these. In some ways less about features, and more about style, I guess 🙂
      Anyhow, in MarkupMenu I've tried to correct those little hiccups, modernise the default markup, and allow for more flexibility with placeholder variables and additional / different options. MarkupMenu was built for ProcessWire 3.0.112+ and with PHP 7.1+ in mind, it's installable with Composer, and I have a few additional ideas (such as conditional placeholders) still on my todo list.
      One more small(ish) difference is that MarkupMenu supports overriding default options via $config->MarkupMenu. I find myself redefining the default markup for every site, which until now meant that each site had a wrapper function for MarkupSimpleNavigation (to avoid code / config repetition), and this way I've been able to leave that out 🙂
      Requirements
      ProcessWire >= 3.0.112 PHP >= 7.1.0 If you're working on an earlier version of ProcessWire or PHP, use MarkupSimpleNavigation instead.
    • By teppo
      Hey folks!
      I'm happy to finally introduce a project I've been working on for quite a while now: it's called Wireframe, and it is an output framework for ProcessWire.
      Note that I'm posting this in the module development area, maily because this project is still in rather early stage. I've built a couple of sites with it myself, and parts of the codebase have been powering some pretty big and complex sites for many years now, but this should still be considered a soft launch 🙂
      --
      Long story short, Wireframe is a module that provides the "backbone" for building sites (and apps) with ProcessWire using an MVC (or perhaps MVVM – one of those three or four letter abbreviations anyway) inspired methodology. You could say that it's an output strategy, but I prefer the term "output framework" since in my mind the word "strategy" means something less tangible. A way of doing things, rather than a tool that actually does things.
      Wireframe (the module) provides a basic implementation for some familiar MVC concepts, such as Controllers and a View layer – the latter of which consists of layouts, partials, and template-specific views. There's no "model" layer, since in this context ProcessWire is the model. As a module Wireframe is actually quite simple – not even nearly the biggest one I've built – but there's still quite a bit of stuff to "get", so I've put together a demo & documentation site for it at https://wireframe-framework.com/.
      In addition to the core module, I'm also working on a couple of site profiles based on it. My current idea is actually to keep the module very light-weight, and implement most of the "opinionated" stuff in site profiles and/or companion modules. For an example MarkupMenu (which I released a while ago) was developed as one of those "companion modules" when I needed a menu module to use on the site profiles.
      Currently there are two public site profiles based on Wireframe:
      site-wireframe-docs is the demo&docs site mentioned above, just with placeholder content replaced with placeholder content. It's not a particularly complex site, but I believe it's still a pretty nice way to dig into the Wireframe module. site-wireframe-boilerplate is a boilerplate (or starter) site profile based on the docs site. This is still very much a work in progress, but essentially I'm trying to build a flexible yet full-featured starter profile you can just grab and start building upon. There will be a proper build process for resources, it will include most of the basic features one tends to need from site to site, etc. --
      Requirements and getting started:
      Wireframe can be installed just like any ProcessWire module. Just clone or download it to your site/modules/ directory and install. It doesn't, though, do a whole lot of stuff on itself – please check out the documentation site for a step-by-step guide on setting up the directory structure, adding the "bootstrap file", etc. You may find it easier to install one of the site profiles mentioned above, but note that this process involves the use of Composer. In the case of the site profiles you can install ProcessWire as usual and download or clone the site profile directory into your setup, but after that you should run "composer install" to get all the dependencies – including the Wireframe module – in place. Hard requirements for Wireframe are ProcessWire 3.0.112 and PHP 7.1+. The codebase is authored with current PHP versions in mind, and while running it on 7.0 may be possible, anything below that definitely won't work. A feature I added just today to the Wireframe module is that in case ProcessWire has write access to your site/templates/ directory, you can use the module settings screen to create the expected directories automatically. Currently that's all, and the module won't – for an example – create Controllers or layouts for you, so you should check out the site profiles for examples on these. (I'm probably going to include some additional helper features in the near future.)
      --
      This project is loosely based on an earlier project called pw-mvc, i.e. the main concepts (such as Controllers and the View layer) are very similar. That being said, Wireframe is a major upgrade in terms of both functionality and architecture: namespaces and autoloader support are now baked in, the codebase requires PHP 7, Controllers are classes extending \Wireframe\Controller (instead of regular "flat" PHP files), implementation based on a module instead of a collection of drop-in files, etc.
      While Wireframe is indeed still in a relatively early stage (0.3.0 was launched today, in case version numbers matter) for the most part I'm happy with the way it works, and likely won't change it too drastically anytime soon – so feel free to give it a try, and if you do, please let me know how it went. I will continue building upon this project, and I am also constantly working on various side projects, such as the site profiles and a few unannounced helper modules.
      I should probably add that while Wireframe is not hard to use, it is more geared towards those interested in "software development" type methodology. With future updates to the module, the site profiles, and the docs I hope to lower the learning curve, but certain level of "developer focus" will remain. Although of course the optimal outcome would be if I could use this project to lure more folks towards that end of the spectrum... 🙂
      --
      Please let me know what you think – and thanks in advance!
    • By Robin S
      After forgetting the class name of the wonderful AdminPageFieldEditLinks module for what feels like the 100th time I decided I needed to give my failing memory a helping hand...
      Autocomplete Module Class Name
      Provides class name autocomplete suggestions for the "Add Module From Directory" and "Add Module From URL" fields at Modules > New.
      Requires ProcessWire >= v3.0.16.
      Screencast

      Installation
      Install the Autocomplete Module Class Name module.
      Configuration
      Add Module From Directory
      Choose the type of autocomplete suggestions list: "Module class names from directory" or "Custom list of module class names". The latter could be useful if you regularly install some modules and would prefer a shorter list of autocomplete suggestions. The list of class names in the modules directory is generated when the Autocomplete Module Class Name module is installed. It doesn't update automatically (because the retrieval of the class names is quite slow), but you can use the button underneath when you want to retrieve an updated list of class names from the directory. Add Module From URL
      If you want to see autocomplete suggestions for the "Add Module From URL" field then enter them in the following format:
      [autocomplete suggestion] > [module ZIP url]
      Example: RepeaterImages > https://github.com/Toutouwai/RepeaterImages/archive/master.zip Awesomplete options
      The "fuzzy search" option uses custom filter and item functions for Awesomplete so that the characters you type just have to exist in the autocomplete suggestion item and occur after preceding matches but do not need to be contiguous. Uncheck this option if you prefer the standard Awesomplete matching. Custom settings for Awesomplete can be entered in the "Awesomplete options" field if needed. See the Awesomplete documentation for more information.  
      https://github.com/Toutouwai/AutocompleteModuleClassName
      https://modules.processwire.com/modules/autocomplete-module-class-name/
×
×
  • Create New...