ryan

This week we’re proud to announce the newest ProcessWire master version 3.0.164. Relative to the previous master version (3.0.148) this version adds a ton of new and useful features and fixes more than 85 issues, with more than 225 commits over a period of 7 months. There were also a lot of technical changes in the core, including some pretty significant refactoring of several classes to further optimize and improve upon our already solid core, making it that much better. I’m confident this master version is one of our best and if you are running an older version I’d encourage you to upgrade when convenient to do so—I think you will really like what this version brings! Read all about what’s new in the latest blog post: https://processwire.com/blog/posts/pw-3.0.164/

@schwarzdesign I was never able to duplicate that one. Pasting in old password makes the new password editable when I test here. But I think this one has come up before and it had something to do with some other admin enhancement module, but I don't remember for sure. There's a reply in that thread that says it only happens of entity encoding textformatter is disabled for the title field. I think it's rare for someone to disable that textformatter on the title field and something I wouldn't usually recommend doing. But I think we can also accommodate the request in that report, just wanted to return to it later when back on the dev branch because it's a change of behavior that I think needs more thorough dev branch testing.

In preparation for the next master version, this week no new features were added, but just like last week, more than a dozen issue reports were resolved. Having focused largely on fixing various issues over the last month, I feel pretty confident that the current dev branch is significantly more solid than the 3.0.148 master version. It adds and improves a whole lot, and also fixes a lot. And to the best of my knowledge, there aren’t any new issues between 3.0.146 and 3.0.164 that haven’t already been fixed. Basically, I don’t think it makes any sense to keep all these updates exclusive to the dev branch any longer, so have merged it to master, today. Consider it a soft launch, as I haven’t made it an official tagged version yet. Maybe I’m shy, but wanted to wait till Monday before Git tagging it and making it official. The master branch has a different audience than the dev branch, and so there’s always that possibility that some new issue will appear that hasn’t on the dev branch, and wouldn’t have. So we’ll let it marinate on the master branch for the weekend before broadcasting it very far. By this time next week, I should have a blog post ready that covers all that’s new in this version, which is 226 commits ahead of the previous master (3.0.148), so there’s a lot to cover. I want to thank you all that have been helping to identify and report issues on GitHub as they come up. Having covered a lot of issue reports over the last month, I can see a lot of effort goes into preparing many of the reports. Your work is appreciated. This month I focused primarily on the reports that I thought were likely to make the most difference to the most people. I also focused on issues that I thought could be accommodated without introducing potentially new issues or new code to test. Of course, not every report could be covered, and there’s always more to do, so I’ll be getting back to it on the dev branch here soon. In addition, I’ve held off on some new things I’ve wanted to add for awhile (a feature freeze of sorts) in preparation for an end-of-month master version. I’m looking forward to outlining all that’s new in next week’s blog post. Until then, thanks for reading and if you get a chance to test out the new 3.0.164 version, please do and let me know how it goes (both master and dev branches are identical right now). I hope you have a great weekend!

@Zeka That's correct. V4 is split out into a bunch of different classes like this, as ProCache has grown quite a bit. It's possible I might not understand what you mean. But ProCache only clears cache files that exist. It knows ahead of time which ones exist because there's a corresponding DB table entry for every cache file. So it shouldn't attempt to find or clear files for any languages that don't have the content available in the cache. Meaning, so long as the page isn't published in the language that it lacks content in, ProCache isn't going to be attempting to write or clear cache files for that page in that language it's not available in (i.e. no extra overhead). But if what you are looking for is more logic in determining whether a page is cleared for a particular language, that clearPage() call in the earlier hook I mentioned does also accept a 'language' option which limits the clear to a particular language. You can also tell it to clear only a particular URL segment, or segments matching a wildcard pattern. Here's the options for the clearPage method: /** * Clear the cache for a specific page * * Default behavior is to clear for all languages, paginations and URL segment variations. * To clear only specific languages, paginations or URL segments, use the options. * * @param Page $page * @param array $options * - `language` (string|int|Language): Clear only this language (default='') * - `urlSegmentStr` (string): Clear only entries matching this URL segment string, wildcards OR regex OK (default='') * - `urlSegments` (array): Clear only entries having any of these URL segments (default=[]) * - `pageNum` (int|bool): Clear only pagination number (i.e. 2), true to clear all paginations, false to clear no paginations (default=0) * - `clearRoot` (bool|null): Clear root index of page path? (default=false when URL segments or paginations requested, true otherwise) * - `rmdir` (bool): Remove directories rather than index files? (default=false) * - `getFiles` (bool): Get array of files that were cleared, rather than a count? (default=false) * @return int|array Quantity or array of files and/or directories that were removed * */ I think we've got all the tools we need to support that kind of logic on the hook side, but ProCache doesn't do that automatically at present. It's a good idea though, I can explore it further.

@Zeka Thanks for the examples. Reading what you write made me realize I've got a similar case too: https://www.tripsite.com/reviews/ — it's just one page, but hundreds (maybe thousands) of URLs underneath are all URL segment powered for 3 levels. What you are asking for would come in handy here for sure. In the hook example below (which is functional) I clear all /reviews/bike-tours/* segments on the /reviews/ page when a page using template "tour" is saved. And when a page using template "boat" is saved, it clears all of the /reviews/boat-[name]/ segments, where [name] is the name of the boat page that was saved. This is going to come in very handy here, and if I understand your case correctly, I think it's what you are looking for also? $wire->addHook('ProCacheStaticClear::clearedPage', function($event) { $page = $event->arguments(0); // Page that was saved/cleared $clearSegment = ''; if($page->template == 'tour') { // clear all /reviews/bike-tours* segments $clearSegment = 'bike-tours*'; } else if($page->template == 'boat') { // clear all /reviews/boat-name/ segments $clearSegment = "boat-{$page->name}*"; } if($clearSegment) { // clear segments below the /reviews/ page $reviews = $event->pages->get('/reviews/'); $event->object->clearPage($reviews, [ 'urlSegmentStr' => $clearSegment ]); } });

@Zeka I don't think we've got hooks specific to what you are trying to do there, but I've pasted in the list of current hooks that ProCache provides for the static caching side of things. Can you expand on your need with an example? It may be something I can add, though want to make sure I fully understand what you are trying to do. // Is caching allowed for given page? Return true or false // Hooks can modify return value to dictate custom caching rules ProCache::allowCacheForPage(Page $page) // Called when entire cache is being cleared // Returns quantity of items cleared ProCache::clearAll() // Called right before a cache file is written // Hooks can modify $content to adjust what is saved in cache file ProCacheStatic::writeCacheFileReady(Page $page, $content, $file) // Called right before a Page is about to have its cache cleared. // Hooks can make it return false to bypass clearing of page. ProCacheStatic::clearPageReady(Page $page) // Called to execute cache clearing behaviors for given Page // Hooks can modify what behaviors are executed ($behaviors argument) // Returns array indexed by behavior name each with a count of files cleared ProCacheStaticBehaviors::executeCacheClearBehaviors(Page $page, array $behaviors)

@JeevanisM Thanks, I have fixed the table salt comment error. The version in the button is on a cache, so it'll update eventually.

ProcessWire 3.0.163 adds a few new $pages hooks (see PW Weekly #323 for details), adds configurable module upload/install options (ProcessModule), and contains many other minor updates, code refactoring and optimizations. But by far, the majority of updates and commits are related to resolving more than a dozen recent issue reports. That will be the focus next week too, as the goal is to have the next master version out by the end of the month, or the first week of August. Priority focus is on any issues that might be bugs introduced between 3.0.148 (previous master) and 3.0.163, as we want to make sure at minimum we aren’t adding any new bugs from one master version to another. Regarding the new configurable module upload options, the intention here is to add additional safety by having the option of locking down the ability to install modules from the admin. As convenient as it is to be able to install and upgrade modules (during development) directly by URL, file upload or directory; the reality is that—depending on the case—it’s also not the safest thing to have on a client’s production site once development is finished. I think it’s best if module installation and upgrades are left to web developers, who are better equipped to resolve any technical issues that might arise during the process. Though it also depends on the installation, which is why I thought it should be configurable. So now you can specify which install options are available, and they can also depend on whether the site is in debug mode or not: $config->moduleInstall = [ // allow install from ProcessWire modules directory? 'directory' => true, // allow install by module file upload? 'upload' => 'debug', // allow install by download from URL? 'download' => 'debug', ]; Above are the current defaults, which can be changed by specifying your own preferred options in /site/config.php. Boolean true means always allowed, boolean false means never allowed, and string “debug” means: allowed if the site is in debug mode. (I’m currently debating on whether the ‘directory’ option should also be ‘debug’ as well.) In addition to these configuration options, the ProcessModule “New” tab now also provides instructions for manual installation of modules. None of us need it I know, but someone new to ProcessWire might see the prior “New” tab and not realize there’s a really simple and safe way to install modules from the file system. So the instructions just seemed to make sense there for consistency. ProCache 4.0 β released Last week I mentioned a new version of ProCache would be coming out this week and version 4.0 of ProCache was released on Wednesday in beta form. It’s available for download now in the ProCache board download thread. This is one of the biggest upgrades for ProCache yet. If you are upgrading from a previous version, read the upgrade instructions in the README.txt file included with it, as it will guide you through some of the new features, and may save you from having to make an update to your .htaccess file. I mentioned much of this in last week's post, but here’s a summary of what’s new in this version of ProCache relative to the previous version: Major refactor of entire module. Now native to ProcessWire 3.x (ProcessWire 3.0.148+ recommended). New .htaccess rules for static cache. New ability to select .htaccess version (v1 or v2, Tweaks tab). New option to specify how trailing slashes are handled (Tweaks tab). Upgrade SCSS compiler from version 0.7.8 to 1.1.1. Add Wikimedia LESS as additional option to Leafo LESS (select which you want on Tweaks tab). Improved per-template lifespan settings. Improved default and per-template behavior settings. New cache clear behavior: Family (parent, siblings, children). New cache clear behavior: References (page that reference saved page). New cache clear override: No-self (skip clearing page that was saved). Per-template behaviors now supports clearing specific pages by ID or selector. Numerous minor fixes and optimizations throughout. Removed direct .htaccess writing ability, replaced with live example file. Lots of new hookable methods for special cases. New “Tests” tab with the following built-in tests: cache function and performance; cache clear behaviors tests; test http response headers; test for GZIP, LZW, zlib/deflate or Brotli compression; Test for keep-alive connection. Thanks for reading and have a great weekend!

Greetings from the sunny covid hotspot state of Georgia, where we haven’t left the house since March. And now getting ready for the kids to start a new school year from home with virtual learning. Everyone delivers everything now, so there’s no need to go out to a grocery store anymore (or go anywhere). I live about a mile from the CDC, so our school district has more kids with parents working at the CDC than any other. That gives me some comfort, knowing that I won’t be sending my kids back to school until the experts at the CDC are willing to; when it’s really and truly safe. Though I don’t think it’s going to be safe for a long, long time. The US is a rudderless ship right now, so we just have to ride it out. Thankfully, we’re all staying safe and keeping busy. The kids are building houses in Roblox (an online game addiction they have), we’ve converted our yard to be a summer camp, and converted the basement to be a gym, while we clear more space to start building out a massive N-scale train set—my 3 locomotives still work perfectly, even after 35 years of storage. And I’ve been learning how to manage chlorine and PH in an inflatable kids pool that keeps the family cool in the hot weather. The kids miss school and other activities, my wife misses being at her office and people she works with, and we all miss our friends and family, but it’s the way things are now, and I’m just grateful to have my immediate family home and safe; and in place where we can ride out the storm. I’m also really glad that I can work on the ProcessWire core and modules for pretty much the entire work day, and enjoying coding as much as I ever have; feeling great about where ProcessWire is and where it’s going, thanks to all of you. I’ve been working on the latest ProCache version the entire week, so not many core updates to report today other than some new hooks added to the Pages class (they are hooks that the new ProCache can use as well). I’d hoped to have this version of ProCache finished by now, but I keep finding more stuff to improve, so decided give it another 2 days of work and testing, and if all looks good, it’ll be ready to release, which will be next week. This version is essentially a major refactor, where just about every line of code has been revisited in some form or another. But if you are already a ProCache user, you’ll also find it very familiar. While I don’t have it posted for download today, below is a brief look at what’s new. Completely new .htaccess rules (v2) that take up a lot less space, especially when using multiple hosts, schemes or extensions. Ability to choose .htaccess version (v1 or v2). ProCache now creates an example .htaccess-procache file that you can rename and use or copy/paste from. ProCache now has a built-in URL testing tool where you can compare the non-cached vs. cached render times. New setting to specify how ProCache delivered URLs should respond to trailing vs. non-trailing slashes in URL. Significant refactor that separates all ProCache functions into separate dedicated classes. Improved custom lifespan settings with predefined template lines. Improved behavior settings with predefined template lines and simpler letter (rather than number) based definitions. Ability to specify predefined cache clearing behaviors, specific pages to clear, or page matching selectors, from within the ProCache admin tool. New predefined cache clearing behavior: Reset cache for family of saved page (parents, siblings, children, grandchildren, and all within). New predefined cache clearing behavior: Reset cache for pages that reference saved page (via Page references). New versions of SCSS and LESS compilers. ProCache is completely ProcessWire 3.x native now (previous versions still supported PW 2.x even if 3.x was recommended). Numerous other improvements, fixes and optimizations throughout. I’ve previously mentioned a built-in crawler in ProCache. That part has been moved to a separate module called ProCacheCrawler and will be released a little later in the ProCache board. It was taking a little too much time to develop, so I didn’t want to hold up the rest of ProCache while I developed that. When installed, ProCache communicates with the crawler, identifying and adding URLs to a queue to be crawled and primed for the cache. What it does is pretty cool already, but it needs more time to develop. It’s also something that depends on being run regularly at intervals (like with CRON) so it’s a little bit of a different setup process than the rest of ProCache, which is another reason why I thought I’d develop is as a separate module. I’ll be working more on finishing development of the crawler later in the year, after the next master version of ProcessWire core is released. Next week I'll have the new ProCache version ready for download as well as a new core version on the development branch. It will focus mostly on fixes for issue reports as we continue working towards the next master version. Thanks for reading and have a great weekend!

Relative to ProcessWire 3.0.161, version 3.0.162 contains 24 commits that continue upgrades/improvements to selector operators, fix various minor issues, add new API convenience methods, improve documentation, optimize and refactor various portions of code and DB queries, and much more. For full details, see the dev branch commit log as well as last week’s post. Next week I hope to finally finish up a new version of ProCache and continue with some additional core to-do items. By early August my hope is that we’ll have the next master branch version ready. Also added this week is a new dedicated documentation page on this site that covers all of ProcessWire’s selector operators, including all the newly added ones here: selector operators. Thanks for reading and have a great weekend!

Good article and postgresql looks interesting with its search capabilities, thanks. Though none of these really solve what I was after here. I experimented quite as bit with stemming and different stemming libraries. Though they all did roughly the same thing. When it came to searching, stemming just wasn’t that useful. WireWordTools originally had a stemming library and methods, and the appropriate fulltext queries included the word stems with wildcards. In the end, it just wasn’t helpful most of the time. And in the few cases where it was worthwhile, it was redundant, though far less thorough, than what we already had with inflection and lemmatisation. So while stemming can have its uses, it’s not even half way there, if trying to build a smart search. Cool nevertheless that they have it built-in apparently. As far as accent support, ranking and fuzzy search, these are all things that MySQL does as well, though maybe there are differences in how they do them. For instance, MySQL supports “sounds like” and also supports pluggable parsers for fulltext searches. Fuzzy search also isn't what I'm after here, but certainly interested in exploring in the future. For me the most useful thing by far is boolean mode searches, particularly in InnoDB, which has a full-text engine modeled on Sphinx. Boolean mode searches are really very powerful, enabling you to specify what’s required, what’s excluded, matching of words or phrases, partial matching of words with wildcards, specifying noise words, isolating distance between words, adjusting ranking up or down on a per-word basis, grouped expressions and nested subexpressions. All while being incredibly fast. I’m pretty thrilled with what MySQL supports here and what it brings to ProcessWire. Postgresql looks very nice too, but for our needs, I don’t feel we are lacking anything relative to it. I think anyone that would say that as a general thing is not very familiar with what MySQL fulltext supports, or maybe is thinking of fulltext support where it was back a long time ago. For ProcessWire and the scale that most use it at, MySQL fulltext is really a sweet spot, enabling PW to deliver enormous power and capability when it comes to search features.

@bernhard I didn't come up with the dictionary words in the JSON files, they are converted from an existing one (here) and apparently the original source is wordnet.princeton.edu. So I'm not sure if those particular words are intended or mistakes. New to me, but "wa" and "wo" are actual English words. Though as far as I can tell they aren't related to "was" or "will". I can't imagine those two instances will ever be helpful for our intended use case so maybe it makes sense to remove them. My plan was to keep looking for more existing dictionaries and continue to merge them into the one in WireWordTools so that it becomes more comprehensive over time.

This week I'm not bumping the version number just yet because I've got lots of work in progress. The biggest thing so far is something I hinted at last week. Basically, I like what the addition of the MySQL query expansion operators have brought (per posts last week and week before), but they also reveal what's lacking: something as simple as a search for "books" still can't directly match the word "book". But that's the most basic example. It's not a limitation of ProcessWire, but just the type of database indexes in general. I think it'd be amazing if ProcessWire had the ability of being really smart about this stuff, able to interpolate not just plurals vs. singulars but related words. In a perfect world, this is what query expansion would do (in addition to what it already does). But the reality is that it involves all kinds of complicated logic, rules and dictionaries; well beyond the scope of even a database. And it can be vastly different depending on the language. So this isn't something we can just add to the core and have it work. On the other hand, I figured maybe we should just put in a hookable method that just pretends the ability was there. Then people could hook it and make it respond with variations of words, according to their needs. The searches that use query expansion could then call this method and use whatever it returns... for when someday the ability is there. So I went ahead and added that hook — WireTextTools::wordAlternates(). And our database-searching class (DatabaseQuerySelectFulltext) now calls upon it, just in case an implementation is available. Well, after getting that hook added and having our class call it, naturally I wanted to test it out. So I got to work on it and came up with this module: WireWordTools. The WireWordTools module provides an API for English word inflection and lemmatisation. And it hooks that new method mentioned above, so that you can install it and immediately have it bring your searches to the next level. While it only helps for English-language searches, maybe we'll be able to add more languages to it, or maybe it'll lead to other modules that do the same thing for other languages. The expanded/alternate words are only used for searches that use the new query expansion operators, which are the ones that have a "+" in them: ~+=, ~|+=, *+=, **+=. They all can return similar results, but are weighted differently. Unlike most operators, where the logic is direct and you can expect them to always behave the same way, these query expansion operators are more subjective, and ones I think we should intend to keep tweaking and improving over time to continually improve the quality of the results they return. Basically, they are geared towards building site search engines, so I think it makes sense for us to pursue anything that makes them better at that, rather than aiming to always have them return the same thing. I am currently testing out the ~|+= operator ("contains any words expand") on our main site search engine here, along with the WireWordTools module. Finally, searching for "books" does match "book" too, and a lot more. More to be done here, but it's a good start hopefully.

Version 3.0.161 on the dev branch continues with the updates optimizing our support for the new selector operators introduced in last week's blog post for 3.0.160. Last week I was still kind of figuring it out and the code still needed some refactoring and optimization. This week several parts have been rewritten and it's been improved quite a bit. Though the end result is still very similar to what was demonstrated last week, but now it's a lot more performant and solid. One thing new this week that's also kind of fun: you can now use more than one operator in any "field=value" selector expression, at least from the API side. And it works anywhere that you might use a selector, whether querying the database or something in memory. I think the best way to explain it is with an example. Let's say that you want to find all pages with a title containing the phrase "hello world" — you'd do this using the "contains text/phrase" operator: *= $pages->find("title*=hello world"); But let's also say that if you don't find any matches, you want to fallback to find any pages that contain the words "hello" and "world" anywhere in the title, in any order. We'd use the "contains all words" operator "~=" to do that. So now you can add that operator to the existing one, and it'll fallback to it if the first operator fails to match. So we'll append the "contains all words" operator to the previous one: ~= $pages->find("title*=~=hello world"); Cool huh? But maybe we still aren't finding any matches, so we want to fallback to something even broader. So if the phrase match fails, and the words match fails, now we want to fallback to find any pages that contain the world "hello" OR "world", rather than requiring them both. For that we can use our new "contains any words" operator: ~|= $pages->find("title*=~=~|=hello world"); This example is getting a bit contrived now, but let's say that if we still haven't found a match, we want it to find any pages that have any words starting with "hello" or "world", so it would find pages with words like "helloes", "worlds", "worldwide", etc. That's a job for our new "contains any partial words" operator: ~|*= $pages->find("title*=~=~|=~|*=hello world"); Okay last one I promise—you probably wouldn't stack this many in real life, but stay with me here. Let's say the query still didn't find anything, and as a final fallback, we want it to find any words LIKE "hello" or "world", so that those terms can match anywhere in words, enabling us to find pages with words like in the previous example, but also words like "phellogen", "othello", "underworld", "otherworldly", etc., and that's a job for our new "contains any words like" operator: ~|%= $pages->find("title*=~=~|=~|*=~|%=hello world"); So that looks like a pretty complex operator there, but as you've seen by following the example, it's just these 5 appended operators to each other: *= Contains phrase ~= Contains all whole words ~|= Contains any whole words ~|*= Contains any partial words ~|%= Contains any words like I think a more likely scenario in a site search is that you might stack two operators, such as the *= followed by the ~|*=, or whatever combination suits your need. By the way, you can do this with any operators, not just the text searching ones. But if you didn't read the blog post last week, also be sure to check out the other new operators in addition to those above: *+=, **=, **+=, ~*=, ~+=, ~~=, ~%=, #=. I think these new operators help out quite a bit with ProcessWire's text searching abilities. But there's one thing that's kind of a common need in search engines that's not easily solved, and that's the handling of singular vs. plural. At least, that's a common issue when it comes to English (I'm assuming so for other languages, but not positive?) MySQL fulltext indexes can't differentiate between singular and plural versions of words, so they index and match them independently as completely different words. This can be unexpected as clients might type in "goose" and expect it's also going to match pages with "geese". I've already got something in the works for this, so stay tuned. Thanks for reading and hope you have a great weekend!

@Robin S That's correct that the combination is not possible for technical reasons. The query expansion is a feature of MySQL fulltext indexes in a standard natural language match/against query, so it's not something that can be used with LIKE (which does not use fulltext index), and also can't be used with match/against boolean mode (which is how partial word matches can use fulltext). So there are only certain operators we can use query expansion with. My experience so far is that query expansion is pretty inconsistent as to when it's useful. But since it's something that MySQL supports I thought PW should provide the option. If a search returns no results, query expansion isn't going to help, because it needs there to be matches in order to expand upon them. I'm currently thinking I might use it on one or two word searches that might otherwise only return a small number of results. If your goal is to bring in as many results as possible I would start experimenting with the "Contains match and expand" **+= operator, which uses the standard MySQL match/against logic, and doesn't require all search words to be present. Though depending on the search, once you go past one or two words with query expansion, it tends to add noise to the results. So I've been thinking a few of these new operators might be made configurable to automatically substitute another operator under certain conditions, like to a non-expand version depending on word count, or automatically substitute another operator when one produces no results.

In ProcessWire 3.0.160 we’ve got some major upgrades and additions to our text-matching selectors and operators. This brings a whole new level of power to $pages->find() and similar API calls, especially when it comes to search engine type queries. This blog post also includes a demo search engine that lets you test everything out live— https://processwire.com/blog/posts/pw-3.0.160/

In the blog post last week we looked at some of the two-factor authentication system upgrades, like the new “remember this computer” feature. This week I finished off the remaining parts, as well as released new versions of both the TfaTotp and TfaEmail modules. Auto-enable TFA support We now have auto-enable support (forced 2FA), which lets you setup two-factor authentication for users, without their input (if they haven’t enabled it already). This is a good way to add a lot of security for very little work. Currently, the module that supports this is the TfaEmail module. That’s because it’s a safe bet to assume the user has access to their email, even if they haven’t specifically setup 2FA. So email is a very good way to nudge people into 2FA, and people are already used to this, as many online services now do it. Considering that the computer can now be remembered, I think it’s unlikely you’ll get any complaints from users. Setting up auto-enable is really simple. Grab the latest version of the TfaEmail module and install it. Then go to your ProcessLogin module settings (Modules > Configure > ProcessLogin) and you’ll see an option there to select Email in the “Force two-factor authentication - Type” field. If you want to limit this to specific roles, then you can also do that here. If you don’t select any roles, then it applies to all roles. Once setup, any user logging into your admin will be asked to enter an authentication code sent to their email, and they’ll need that code to complete the login. Chances are they’ll also click that “remember this computer” checkbox so that they can skip the code on future logins. TfaEmail version 2 The new version of the TfaEmail module also lets you now configure what WireMail module you want it to use for sending authentication emails. If using multiple mail sending services, you’ll want your most reliable and fastest email sending service to handle these kinds of transactional emails. TfaTotp version 4 Once users understand the benefits of 2FA, chances are they’ll want to upgrade to TOTP, where they can use a dedicated authenticator app. The ProcessWire TfaTotp module got several upgrades this week. The biggest was the addition of a locally hosted QR code generator (QRCode for PHP by Kazuhiko Arase). No longer does it have to rely upon an external service to generate QR codes (previous versions used Google Charts for QR code generation). In addition, the TOTP TwoFactorAuth library has been updated to the latest version. Moving those modules to the core Speaking of those two modules (TfaEmail and TfaTotp), thanks for your input last week about their inclusion in the core. It sounds like most think it’s a good idea, so I think we’ll go that route. But I need a little more time to do that, so going to hold that update and the 3.0.160 version bump for next week. Coming next week: Useful new selector operators Next week I’ve also got a couple of special new text-matching operators being added to our selectors system that I think you are going to really like. They are operators that are especially useful to those building text search engines, and ones that I’ve found so useful this week that I wish we’d had them since the beginning. I’m excited to add those into 3.0.160 and tell you more about them next week. By the way, while 3.0.160 isn't officially the version on the dev branch yet, if you download the current dev branch version (3.0.159), all of the TFA updates mentioned above are present and ready to use.

Finally, a blog post this time ? — ProcessWire 3.0.159 brings some useful and time-saving upgrades to the core two-factor authentication system: https://processwire.com/blog/posts/pw-3.0.159/

There were lots of core updates this week, very much in the same theme as last week — major long term quality improvements for the core, but no shiny toys to play with… Just lots of good and solid foundational core improvements, and a few fixes too. It's mostly kind of technical stuff that's probably not that interesting to read about, so I'll keep this short, but for those interested here's the commit log. Thanks for reading and have a great weekend!

@HMCB Not a lot left to be done for this particular update. Mostly just details as I come across them. For instance, this morning I noticed that FieldtypeMulti needs some of the same updates I applied to Fieldtype, but nothing that's going to change the way it works or how it behaves. So while not particularly urgent, I just want to cover all the related updates while still fresh on the mind.

ProcessWire 3.0.157 on the development branch continues the trend of core refactoring that’s been happening quite a bit in 2020. Rather than doing a rewrite every few years (like some CMS projects) we instead refactor parts as we go, constantly improving and optimizing the core. This works because the core design/architecture is right where it needs to be, even 10 years in. But there’s always still bits of legacy code, and code that can be improved. So in the context of ProcessWire, refactoring means incrementally rewriting code on the inside, without changing its behavior on the outside (other than making it faster and/or more secure). This has been happening regularly over the last 10 years, and will likewise continue happening over the next 10 years and likely beyond. This week the code behind ProcessWire’s core Database and PageFinder classes got a major refactoring. This is some of the most used code in PW, as it handles everything involved in taking a selector and converting it to a database query. But it’s always been a little bit of a pain point for me because it had to build queries in a way that I thought wasn’t ideal, in order to make it possible for lots of different modular parts (mostly Fieldtype modules) to contribute to the query and for PageFinder to put it all together. It was fast and secure, but still one of those parts that felt like a little too much duct tape to me. But considering how crucial the code is, I’ve always been reluctant to make major changes, since it all worked just fine. Spending lots of years thinking about it (on and off), a desire to work out any pain points, and having better tools available (like Phpstorm and Tracy) made it possible to finally massage out this pain point. Some work still remains to be done, but it’s mostly there and I’m feeling good about it. Stuff like this is key for the maintenance and longevity of the core, and involved a lot of time and effort, but makes very little difference to users, even if it makes a lot of difference to me in maintaining the core. It would make a boring blog post for sure—lots of work and changes, but no new toys to show for it. Nevertheless, it has that feeling of a good house cleaning, even if you can't see it from the outside. The scope of changes here means that there may actually be new bugs to work out, so to be on the safe side, consider 3.0.157 to be a little more “beta” than the dev branches usually are. Though I’m running it here on processwire.com and it’s working well. Beyond the fairly major updates to the Database classes, there are also a few new Sanitizer convenience methods that are primarily variations on existing ones, but useful ones for sure. Thanks for reading and have a great weekend!

This week I was back to focusing on the core and got quite a lot done. A lot of GitHub issue reports were resolved, plus several minor tweaks and additions were made in 3.0.156 as well. But the biggest update was the addition of the $pages->parents() API, which is something that I think you’ll likely have zero use for (and why I’m not putting it into a blog post) but something that the core itself will use quite a bit, and is a really nice improvement for the system and its scalability. So if you don’t mind some technical reading, read on. Whenever you call a $page->find() method ($page, not $pages) or use a “has_parent=“ in a selector, ProcessWire joins in a special table for the purpose called pages_parents. It uses this table to keep track of family relationships that aren’t otherwise apparent. For instance, let’s say we have page “g” that lives at path /a/b/c/d/e/f/g/. Page “g” only knows that it has page “f” as its parent. It doesn’t know that page “e” is its grandparent unless or until page “f” is loaded. Once “f” is loaded, then “f” can reveal its parent “e”. It works the same for every relationship down to the root parent “a”. So the pages are like a linked list or blockchain of sorts, where only 1 relationship forward or backward is known per page. The “pages_parents” table fills in this gap, enabling PW to quickly identify these relationships without having to load all the pages in the family. This is particularly useful in performing find() operations that you want to limit to a branch started by a particular parent. It’s the reason why we have both $pages->find() that searches the entire site, and $page->find() that limits the search within the branch started by $page. I haven’t paid much attention to the code behind this pages_parents table because it generally just worked, needing little attention. But I came across a couple of cases where the data in the table wasn’t fully accurate with the page tree, without a clear reason why. Then I became aware of one large scale case from a PW user where it was a huge bottleneck. It involved a large site (250k+ pages) and a recursive clone operation that appeared to involve hundreds of pages. But that operation was taking an unreasonable 10 minutes, so something wasn’t right. It came down to something going on with the pages_parents table. Once I dove into trying to figure out what was going on, I realized that if I was to have any chance of keeping track of it, we needed a dedicated API for managing these relationships and the table that keeps track of them. So that’s what got a lot of attention this week. While still testing here, it does appear initially that the 10 minute clone time has gone down to a few seconds, and everything about this relationship management is now rewritten, optimized and significantly improved. It was a lot of work, but absolutely worth it for PW. Rebuilding the entire table from scratch now takes between 2-3 seconds on a site with 250k pages and 150k relationships. The new API can be accessed from $pages->parents(). This API is really useful to the work that I do here (maintaining the core) but I’ll be honest, it’s probably not useful to most others, so I won’t go into the details here, other than to say I’m happy with it. But if you are interested, there are methods finding all the parents in a site, or a particular branch, and methods for rebuilding the pages_parents table, among others. Maybe more will be added to it later that actually would be useful in the public API, but for now I’ll likely leave it out of our public API docs. The $sanitizer->selectorValue() method also got a full rewrite this week (actually, one the last few weeks). It’s now quite a bit more comprehensive and configurable (see the new method options). The previous version was just fine, and actually still remains — you can use it by specifying [ ‘version’ => 1 ] in the $options argument to the selectorValue() method. But the new version is coded better and covers more edge cases, plus provides a lot more configurability for the times when you need it.

Beluga, I'm not much into carousels either, but also wouldn't claim there's no place for them. There's 1 small carousel on this entire site, and there was also one on the previous iteration of the site—the client has always liked it, and the customers react well to it. I really like this client for a lot of reasons, but one is that they are much more involved than most, know their technology, their product and their audience better than anyone I've worked with. The carousel is not my idea, but I trust and am certain the client knows their customers better than any self proclaimed experts online. I got a kick out of that linked anti-carousel site because it's a bit of a self own by whoever made it—it uses a carousel to make points that we likely would not have bothered to read if they weren't in a carousel. ?

Last week I told you guys about how I was working on development of a client’s site and deep diving into the details. Well this week was the same, except perhaps more so. Yesterday we finally launched phase one. There’s still more to do, and still working out some small bugs and such. This is a site I’ve been developing since back in the early versions of ProcessWire 2.x, which I think was nearly a decade ago. In fact, this was one of the first sites running ProcessWire 2.x, if I recall correctly. We’ve been keeping it up-to-date over the years, but this is the first time we’ve done a full front-end redo of the site, essentially starting from scratch on that, and spending a few months on it. But the admin side (fields, templates and page structure) remains roughly the same from where it was 10 years ago, and that’s what we’re going to redo in phase 2 (this year). There’s a lot of fields in this site, so I’m looking forward to really simplifying it with ProFields, repeaters and more — none of these existed when the original site was built. One thing really great about this iteration of the site is that it’s a ProcessWire community collaboration. Pete (our awesome forum administrator) did the design, as well as most of the front-end markup/css for the site. Jan (our awesome system administrator) setup the servers that it runs on, with AWS load balancer setup and everything related to that (he’s also one of the owners of the site). I did the other development stuff (the ProcessWire API side of things), making all the content fill all the markup in the right places, structure and organization, markup regions, search engines, optimization, etc., basically the stuff needed to get it all working in PW. This is the first time that I’ve developed a site where we can run new and old site side-by-side off the same ProcessWire installation. During development, the client could click a checkbox in their user profile and then they’d be using the new site. Behind the scenes, it does this using a fairly new feature in ProcessWire which is: $config->setLocation(‘templates’, ‘site/tpl’); So we had site/tpl/ having the new site templates, while site/templates/ had the old version. So one limitation for this phase 1 is that the old and new sites had to deliver the same exact content, even if the resulting output was very different. This site also uses custom page classes (another new feature), Markup Regions, and ProcessWire’s Functions API, and the Uikit 3 front-end framework. Pete also wrote a nice custom PW module for this site to handle localized weather forecasts — I ran out of time to get it in place yesterday, but that should be there next week hopefully. Yesterday we launched the site and we were finally able to start running it through its paces with live traffic. I thought I was going to be working on the PW core today, but you know that as soon as you launch a new site you always find things that need adjustment and can’t wait, so that was all of today. There’s more optimization work to do, and then there’s phase 2, where we start using ProFields and Repeaters to better isolate a lot of data and be able to implement the rest of Pete’s design in the parts that we weren’t able to in phase 1. This is where things like the current pricing tables (one example of a weak point) start to look really sharp. But I’m also looking forward to taking a little breather and catch up on some serious PW core work, issue reports and module updates over the next few weeks before diving into phase 2 of this site. I didn’t want to share the site quite yet because there’s still some details to take care of and such. But here it is Friday and I’ve got no other ProcessWire news to report, so was feeling a little guilty that I didn’t get more ProcessWire core work done over the last week, due to being focused on developing this site. But this was one of the first sites running ProcessWire, so it's an important one to me. And here it is about 10 years later, still the same installation (templates, fields, page tree) but with a brand new design and running the latest PW, and lots more to come. Tripsite.com

I hope that you all are having a good week and staying healthy. I’ve been very much focused on preparing a new ProcessWire-powered client’s website for launch, hopefully sometime next week. This is a site that’s already pretty well established with the search engines and such, so I’ve been going through all of the pre-launch QA in terms of optimization, SEO accessibility, mobile testing, getting the multi-language translators everything they need to translate, adding ProCache to the picture, and so on. For me this is one of the most fun parts of development, with relentless optimization and having it all come together… but also one of the most time intensive. This is where all the little details surface, things that need improvement become more apparent, and new ideas keep popping up. It’s also where lots of the bugs are discovered and fixed, and where I sometimes have to slap myself on the head for missing one thing or another, but then of course taking care of it. So that’s been most of this week, and I think likely at least half of next week too. But then it’s back to work on the core and modules, covering issue reports and more. I’ve actually been at work on some parts of the core and ProCache this week as well, but not to the point of finishing anything major. So you’ll see some updates on the core this week, and there’s also lots of stuff pending that’s not yet committed, and it’s all work in progress, so stuff to review later. With kids at home I’m still moving at a little slower pace than usual, but starting to get the hang of it, doing well and building momentum. While I don’t have much new to offer in this update, thanks for reading anyway, and I hope that you have a great weekend!