ryan

@Zeka I don't think we've got hooks specific to what you are trying to do there, but I've pasted in the list of current hooks that ProCache provides for the static caching side of things. Can you expand on your need with an example? It may be something I can add, though want to make sure I fully understand what you are trying to do. // Is caching allowed for given page? Return true or false // Hooks can modify return value to dictate custom caching rules ProCache::allowCacheForPage(Page $page) // Called when entire cache is being cleared // Returns quantity of items cleared ProCache::clearAll() // Called right before a cache file is written // Hooks can modify $content to adjust what is saved in cache file ProCacheStatic::writeCacheFileReady(Page $page, $content, $file) // Called right before a Page is about to have its cache cleared. // Hooks can make it return false to bypass clearing of page. ProCacheStatic::clearPageReady(Page $page) // Called to execute cache clearing behaviors for given Page // Hooks can modify what behaviors are executed ($behaviors argument) // Returns array indexed by behavior name each with a count of files cleared ProCacheStaticBehaviors::executeCacheClearBehaviors(Page $page, array $behaviors)

@JeevanisM Thanks, I have fixed the table salt comment error. The version in the button is on a cache, so it'll update eventually.

ProcessWire 3.0.163 adds a few new $pages hooks (see PW Weekly #323 for details), adds configurable module upload/install options (ProcessModule), and contains many other minor updates, code refactoring and optimizations. But by far, the majority of updates and commits are related to resolving more than a dozen recent issue reports. That will be the focus next week too, as the goal is to have the next master version out by the end of the month, or the first week of August. Priority focus is on any issues that might be bugs introduced between 3.0.148 (previous master) and 3.0.163, as we want to make sure at minimum we aren’t adding any new bugs from one master version to another. Regarding the new configurable module upload options, the intention here is to add additional safety by having the option of locking down the ability to install modules from the admin. As convenient as it is to be able to install and upgrade modules (during development) directly by URL, file upload or directory; the reality is that—depending on the case—it’s also not the safest thing to have on a client’s production site once development is finished. I think it’s best if module installation and upgrades are left to web developers, who are better equipped to resolve any technical issues that might arise during the process. Though it also depends on the installation, which is why I thought it should be configurable. So now you can specify which install options are available, and they can also depend on whether the site is in debug mode or not: $config->moduleInstall = [ // allow install from ProcessWire modules directory? 'directory' => true, // allow install by module file upload? 'upload' => 'debug', // allow install by download from URL? 'download' => 'debug', ]; Above are the current defaults, which can be changed by specifying your own preferred options in /site/config.php. Boolean true means always allowed, boolean false means never allowed, and string “debug” means: allowed if the site is in debug mode. (I’m currently debating on whether the ‘directory’ option should also be ‘debug’ as well.) In addition to these configuration options, the ProcessModule “New” tab now also provides instructions for manual installation of modules. None of us need it I know, but someone new to ProcessWire might see the prior “New” tab and not realize there’s a really simple and safe way to install modules from the file system. So the instructions just seemed to make sense there for consistency. ProCache 4.0 β released Last week I mentioned a new version of ProCache would be coming out this week and version 4.0 of ProCache was released on Wednesday in beta form. It’s available for download now in the ProCache board download thread. This is one of the biggest upgrades for ProCache yet. If you are upgrading from a previous version, read the upgrade instructions in the README.txt file included with it, as it will guide you through some of the new features, and may save you from having to make an update to your .htaccess file. I mentioned much of this in last week's post, but here’s a summary of what’s new in this version of ProCache relative to the previous version: Major refactor of entire module. Now native to ProcessWire 3.x (ProcessWire 3.0.148+ recommended). New .htaccess rules for static cache. New ability to select .htaccess version (v1 or v2, Tweaks tab). New option to specify how trailing slashes are handled (Tweaks tab). Upgrade SCSS compiler from version 0.7.8 to 1.1.1. Add Wikimedia LESS as additional option to Leafo LESS (select which you want on Tweaks tab). Improved per-template lifespan settings. Improved default and per-template behavior settings. New cache clear behavior: Family (parent, siblings, children). New cache clear behavior: References (page that reference saved page). New cache clear override: No-self (skip clearing page that was saved). Per-template behaviors now supports clearing specific pages by ID or selector. Numerous minor fixes and optimizations throughout. Removed direct .htaccess writing ability, replaced with live example file. Lots of new hookable methods for special cases. New “Tests” tab with the following built-in tests: cache function and performance; cache clear behaviors tests; test http response headers; test for GZIP, LZW, zlib/deflate or Brotli compression; Test for keep-alive connection. Thanks for reading and have a great weekend!

Greetings from the sunny covid hotspot state of Georgia, where we haven’t left the house since March. And now getting ready for the kids to start a new school year from home with virtual learning. Everyone delivers everything now, so there’s no need to go out to a grocery store anymore (or go anywhere). I live about a mile from the CDC, so our school district has more kids with parents working at the CDC than any other. That gives me some comfort, knowing that I won’t be sending my kids back to school until the experts at the CDC are willing to; when it’s really and truly safe. Though I don’t think it’s going to be safe for a long, long time. The US is a rudderless ship right now, so we just have to ride it out. Thankfully, we’re all staying safe and keeping busy. The kids are building houses in Roblox (an online game addiction they have), we’ve converted our yard to be a summer camp, and converted the basement to be a gym, while we clear more space to start building out a massive N-scale train set—my 3 locomotives still work perfectly, even after 35 years of storage. And I’ve been learning how to manage chlorine and PH in an inflatable kids pool that keeps the family cool in the hot weather. The kids miss school and other activities, my wife misses being at her office and people she works with, and we all miss our friends and family, but it’s the way things are now, and I’m just grateful to have my immediate family home and safe; and in place where we can ride out the storm. I’m also really glad that I can work on the ProcessWire core and modules for pretty much the entire work day, and enjoying coding as much as I ever have; feeling great about where ProcessWire is and where it’s going, thanks to all of you. I’ve been working on the latest ProCache version the entire week, so not many core updates to report today other than some new hooks added to the Pages class (they are hooks that the new ProCache can use as well). I’d hoped to have this version of ProCache finished by now, but I keep finding more stuff to improve, so decided give it another 2 days of work and testing, and if all looks good, it’ll be ready to release, which will be next week. This version is essentially a major refactor, where just about every line of code has been revisited in some form or another. But if you are already a ProCache user, you’ll also find it very familiar. While I don’t have it posted for download today, below is a brief look at what’s new. Completely new .htaccess rules (v2) that take up a lot less space, especially when using multiple hosts, schemes or extensions. Ability to choose .htaccess version (v1 or v2). ProCache now creates an example .htaccess-procache file that you can rename and use or copy/paste from. ProCache now has a built-in URL testing tool where you can compare the non-cached vs. cached render times. New setting to specify how ProCache delivered URLs should respond to trailing vs. non-trailing slashes in URL. Significant refactor that separates all ProCache functions into separate dedicated classes. Improved custom lifespan settings with predefined template lines. Improved behavior settings with predefined template lines and simpler letter (rather than number) based definitions. Ability to specify predefined cache clearing behaviors, specific pages to clear, or page matching selectors, from within the ProCache admin tool. New predefined cache clearing behavior: Reset cache for family of saved page (parents, siblings, children, grandchildren, and all within). New predefined cache clearing behavior: Reset cache for pages that reference saved page (via Page references). New versions of SCSS and LESS compilers. ProCache is completely ProcessWire 3.x native now (previous versions still supported PW 2.x even if 3.x was recommended). Numerous other improvements, fixes and optimizations throughout. I’ve previously mentioned a built-in crawler in ProCache. That part has been moved to a separate module called ProCacheCrawler and will be released a little later in the ProCache board. It was taking a little too much time to develop, so I didn’t want to hold up the rest of ProCache while I developed that. When installed, ProCache communicates with the crawler, identifying and adding URLs to a queue to be crawled and primed for the cache. What it does is pretty cool already, but it needs more time to develop. It’s also something that depends on being run regularly at intervals (like with CRON) so it’s a little bit of a different setup process than the rest of ProCache, which is another reason why I thought I’d develop is as a separate module. I’ll be working more on finishing development of the crawler later in the year, after the next master version of ProcessWire core is released. Next week I'll have the new ProCache version ready for download as well as a new core version on the development branch. It will focus mostly on fixes for issue reports as we continue working towards the next master version. Thanks for reading and have a great weekend!

Relative to ProcessWire 3.0.161, version 3.0.162 contains 24 commits that continue upgrades/improvements to selector operators, fix various minor issues, add new API convenience methods, improve documentation, optimize and refactor various portions of code and DB queries, and much more. For full details, see the dev branch commit log as well as last week’s post. Next week I hope to finally finish up a new version of ProCache and continue with some additional core to-do items. By early August my hope is that we’ll have the next master branch version ready. Also added this week is a new dedicated documentation page on this site that covers all of ProcessWire’s selector operators, including all the newly added ones here: selector operators. Thanks for reading and have a great weekend!

Good article and postgresql looks interesting with its search capabilities, thanks. Though none of these really solve what I was after here. I experimented quite as bit with stemming and different stemming libraries. Though they all did roughly the same thing. When it came to searching, stemming just wasn’t that useful. WireWordTools originally had a stemming library and methods, and the appropriate fulltext queries included the word stems with wildcards. In the end, it just wasn’t helpful most of the time. And in the few cases where it was worthwhile, it was redundant, though far less thorough, than what we already had with inflection and lemmatisation. So while stemming can have its uses, it’s not even half way there, if trying to build a smart search. Cool nevertheless that they have it built-in apparently. As far as accent support, ranking and fuzzy search, these are all things that MySQL does as well, though maybe there are differences in how they do them. For instance, MySQL supports “sounds like” and also supports pluggable parsers for fulltext searches. Fuzzy search also isn't what I'm after here, but certainly interested in exploring in the future. For me the most useful thing by far is boolean mode searches, particularly in InnoDB, which has a full-text engine modeled on Sphinx. Boolean mode searches are really very powerful, enabling you to specify what’s required, what’s excluded, matching of words or phrases, partial matching of words with wildcards, specifying noise words, isolating distance between words, adjusting ranking up or down on a per-word basis, grouped expressions and nested subexpressions. All while being incredibly fast. I’m pretty thrilled with what MySQL supports here and what it brings to ProcessWire. Postgresql looks very nice too, but for our needs, I don’t feel we are lacking anything relative to it. I think anyone that would say that as a general thing is not very familiar with what MySQL fulltext supports, or maybe is thinking of fulltext support where it was back a long time ago. For ProcessWire and the scale that most use it at, MySQL fulltext is really a sweet spot, enabling PW to deliver enormous power and capability when it comes to search features.

@bernhard I didn't come up with the dictionary words in the JSON files, they are converted from an existing one (here) and apparently the original source is wordnet.princeton.edu. So I'm not sure if those particular words are intended or mistakes. New to me, but "wa" and "wo" are actual English words. Though as far as I can tell they aren't related to "was" or "will". I can't imagine those two instances will ever be helpful for our intended use case so maybe it makes sense to remove them. My plan was to keep looking for more existing dictionaries and continue to merge them into the one in WireWordTools so that it becomes more comprehensive over time.

This week I'm not bumping the version number just yet because I've got lots of work in progress. The biggest thing so far is something I hinted at last week. Basically, I like what the addition of the MySQL query expansion operators have brought (per posts last week and week before), but they also reveal what's lacking: something as simple as a search for "books" still can't directly match the word "book". But that's the most basic example. It's not a limitation of ProcessWire, but just the type of database indexes in general. I think it'd be amazing if ProcessWire had the ability of being really smart about this stuff, able to interpolate not just plurals vs. singulars but related words. In a perfect world, this is what query expansion would do (in addition to what it already does). But the reality is that it involves all kinds of complicated logic, rules and dictionaries; well beyond the scope of even a database. And it can be vastly different depending on the language. So this isn't something we can just add to the core and have it work. On the other hand, I figured maybe we should just put in a hookable method that just pretends the ability was there. Then people could hook it and make it respond with variations of words, according to their needs. The searches that use query expansion could then call this method and use whatever it returns... for when someday the ability is there. So I went ahead and added that hook — WireTextTools::wordAlternates(). And our database-searching class (DatabaseQuerySelectFulltext) now calls upon it, just in case an implementation is available. Well, after getting that hook added and having our class call it, naturally I wanted to test it out. So I got to work on it and came up with this module: WireWordTools. The WireWordTools module provides an API for English word inflection and lemmatisation. And it hooks that new method mentioned above, so that you can install it and immediately have it bring your searches to the next level. While it only helps for English-language searches, maybe we'll be able to add more languages to it, or maybe it'll lead to other modules that do the same thing for other languages. The expanded/alternate words are only used for searches that use the new query expansion operators, which are the ones that have a "+" in them: ~+=, ~|+=, *+=, **+=. They all can return similar results, but are weighted differently. Unlike most operators, where the logic is direct and you can expect them to always behave the same way, these query expansion operators are more subjective, and ones I think we should intend to keep tweaking and improving over time to continually improve the quality of the results they return. Basically, they are geared towards building site search engines, so I think it makes sense for us to pursue anything that makes them better at that, rather than aiming to always have them return the same thing. I am currently testing out the ~|+= operator ("contains any words expand") on our main site search engine here, along with the WireWordTools module. Finally, searching for "books" does match "book" too, and a lot more. More to be done here, but it's a good start hopefully.

Version 3.0.161 on the dev branch continues with the updates optimizing our support for the new selector operators introduced in last week's blog post for 3.0.160. Last week I was still kind of figuring it out and the code still needed some refactoring and optimization. This week several parts have been rewritten and it's been improved quite a bit. Though the end result is still very similar to what was demonstrated last week, but now it's a lot more performant and solid. One thing new this week that's also kind of fun: you can now use more than one operator in any "field=value" selector expression, at least from the API side. And it works anywhere that you might use a selector, whether querying the database or something in memory. I think the best way to explain it is with an example. Let's say that you want to find all pages with a title containing the phrase "hello world" — you'd do this using the "contains text/phrase" operator: *= $pages->find("title*=hello world"); But let's also say that if you don't find any matches, you want to fallback to find any pages that contain the words "hello" and "world" anywhere in the title, in any order. We'd use the "contains all words" operator "~=" to do that. So now you can add that operator to the existing one, and it'll fallback to it if the first operator fails to match. So we'll append the "contains all words" operator to the previous one: ~= $pages->find("title*=~=hello world"); Cool huh? But maybe we still aren't finding any matches, so we want to fallback to something even broader. So if the phrase match fails, and the words match fails, now we want to fallback to find any pages that contain the world "hello" OR "world", rather than requiring them both. For that we can use our new "contains any words" operator: ~|= $pages->find("title*=~=~|=hello world"); This example is getting a bit contrived now, but let's say that if we still haven't found a match, we want it to find any pages that have any words starting with "hello" or "world", so it would find pages with words like "helloes", "worlds", "worldwide", etc. That's a job for our new "contains any partial words" operator: ~|*= $pages->find("title*=~=~|=~|*=hello world"); Okay last one I promise—you probably wouldn't stack this many in real life, but stay with me here. Let's say the query still didn't find anything, and as a final fallback, we want it to find any words LIKE "hello" or "world", so that those terms can match anywhere in words, enabling us to find pages with words like in the previous example, but also words like "phellogen", "othello", "underworld", "otherworldly", etc., and that's a job for our new "contains any words like" operator: ~|%= $pages->find("title*=~=~|=~|*=~|%=hello world"); So that looks like a pretty complex operator there, but as you've seen by following the example, it's just these 5 appended operators to each other: *= Contains phrase ~= Contains all whole words ~|= Contains any whole words ~|*= Contains any partial words ~|%= Contains any words like I think a more likely scenario in a site search is that you might stack two operators, such as the *= followed by the ~|*=, or whatever combination suits your need. By the way, you can do this with any operators, not just the text searching ones. But if you didn't read the blog post last week, also be sure to check out the other new operators in addition to those above: *+=, **=, **+=, ~*=, ~+=, ~~=, ~%=, #=. I think these new operators help out quite a bit with ProcessWire's text searching abilities. But there's one thing that's kind of a common need in search engines that's not easily solved, and that's the handling of singular vs. plural. At least, that's a common issue when it comes to English (I'm assuming so for other languages, but not positive?) MySQL fulltext indexes can't differentiate between singular and plural versions of words, so they index and match them independently as completely different words. This can be unexpected as clients might type in "goose" and expect it's also going to match pages with "geese". I've already got something in the works for this, so stay tuned. Thanks for reading and hope you have a great weekend!

@Robin S That's correct that the combination is not possible for technical reasons. The query expansion is a feature of MySQL fulltext indexes in a standard natural language match/against query, so it's not something that can be used with LIKE (which does not use fulltext index), and also can't be used with match/against boolean mode (which is how partial word matches can use fulltext). So there are only certain operators we can use query expansion with. My experience so far is that query expansion is pretty inconsistent as to when it's useful. But since it's something that MySQL supports I thought PW should provide the option. If a search returns no results, query expansion isn't going to help, because it needs there to be matches in order to expand upon them. I'm currently thinking I might use it on one or two word searches that might otherwise only return a small number of results. If your goal is to bring in as many results as possible I would start experimenting with the "Contains match and expand" **+= operator, which uses the standard MySQL match/against logic, and doesn't require all search words to be present. Though depending on the search, once you go past one or two words with query expansion, it tends to add noise to the results. So I've been thinking a few of these new operators might be made configurable to automatically substitute another operator under certain conditions, like to a non-expand version depending on word count, or automatically substitute another operator when one produces no results.

In ProcessWire 3.0.160 we’ve got some major upgrades and additions to our text-matching selectors and operators. This brings a whole new level of power to $pages->find() and similar API calls, especially when it comes to search engine type queries. This blog post also includes a demo search engine that lets you test everything out live— https://processwire.com/blog/posts/pw-3.0.160/

In the blog post last week we looked at some of the two-factor authentication system upgrades, like the new “remember this computer” feature. This week I finished off the remaining parts, as well as released new versions of both the TfaTotp and TfaEmail modules. Auto-enable TFA support We now have auto-enable support (forced 2FA), which lets you setup two-factor authentication for users, without their input (if they haven’t enabled it already). This is a good way to add a lot of security for very little work. Currently, the module that supports this is the TfaEmail module. That’s because it’s a safe bet to assume the user has access to their email, even if they haven’t specifically setup 2FA. So email is a very good way to nudge people into 2FA, and people are already used to this, as many online services now do it. Considering that the computer can now be remembered, I think it’s unlikely you’ll get any complaints from users. Setting up auto-enable is really simple. Grab the latest version of the TfaEmail module and install it. Then go to your ProcessLogin module settings (Modules > Configure > ProcessLogin) and you’ll see an option there to select Email in the “Force two-factor authentication - Type” field. If you want to limit this to specific roles, then you can also do that here. If you don’t select any roles, then it applies to all roles. Once setup, any user logging into your admin will be asked to enter an authentication code sent to their email, and they’ll need that code to complete the login. Chances are they’ll also click that “remember this computer” checkbox so that they can skip the code on future logins. TfaEmail version 2 The new version of the TfaEmail module also lets you now configure what WireMail module you want it to use for sending authentication emails. If using multiple mail sending services, you’ll want your most reliable and fastest email sending service to handle these kinds of transactional emails. TfaTotp version 4 Once users understand the benefits of 2FA, chances are they’ll want to upgrade to TOTP, where they can use a dedicated authenticator app. The ProcessWire TfaTotp module got several upgrades this week. The biggest was the addition of a locally hosted QR code generator (QRCode for PHP by Kazuhiko Arase). No longer does it have to rely upon an external service to generate QR codes (previous versions used Google Charts for QR code generation). In addition, the TOTP TwoFactorAuth library has been updated to the latest version. Moving those modules to the core Speaking of those two modules (TfaEmail and TfaTotp), thanks for your input last week about their inclusion in the core. It sounds like most think it’s a good idea, so I think we’ll go that route. But I need a little more time to do that, so going to hold that update and the 3.0.160 version bump for next week. Coming next week: Useful new selector operators Next week I’ve also got a couple of special new text-matching operators being added to our selectors system that I think you are going to really like. They are operators that are especially useful to those building text search engines, and ones that I’ve found so useful this week that I wish we’d had them since the beginning. I’m excited to add those into 3.0.160 and tell you more about them next week. By the way, while 3.0.160 isn't officially the version on the dev branch yet, if you download the current dev branch version (3.0.159), all of the TFA updates mentioned above are present and ready to use.

Finally, a blog post this time ? — ProcessWire 3.0.159 brings some useful and time-saving upgrades to the core two-factor authentication system: https://processwire.com/blog/posts/pw-3.0.159/

There were lots of core updates this week, very much in the same theme as last week — major long term quality improvements for the core, but no shiny toys to play with… Just lots of good and solid foundational core improvements, and a few fixes too. It's mostly kind of technical stuff that's probably not that interesting to read about, so I'll keep this short, but for those interested here's the commit log. Thanks for reading and have a great weekend!

@HMCB Not a lot left to be done for this particular update. Mostly just details as I come across them. For instance, this morning I noticed that FieldtypeMulti needs some of the same updates I applied to Fieldtype, but nothing that's going to change the way it works or how it behaves. So while not particularly urgent, I just want to cover all the related updates while still fresh on the mind.

ProcessWire 3.0.157 on the development branch continues the trend of core refactoring that’s been happening quite a bit in 2020. Rather than doing a rewrite every few years (like some CMS projects) we instead refactor parts as we go, constantly improving and optimizing the core. This works because the core design/architecture is right where it needs to be, even 10 years in. But there’s always still bits of legacy code, and code that can be improved. So in the context of ProcessWire, refactoring means incrementally rewriting code on the inside, without changing its behavior on the outside (other than making it faster and/or more secure). This has been happening regularly over the last 10 years, and will likewise continue happening over the next 10 years and likely beyond. This week the code behind ProcessWire’s core Database and PageFinder classes got a major refactoring. This is some of the most used code in PW, as it handles everything involved in taking a selector and converting it to a database query. But it’s always been a little bit of a pain point for me because it had to build queries in a way that I thought wasn’t ideal, in order to make it possible for lots of different modular parts (mostly Fieldtype modules) to contribute to the query and for PageFinder to put it all together. It was fast and secure, but still one of those parts that felt like a little too much duct tape to me. But considering how crucial the code is, I’ve always been reluctant to make major changes, since it all worked just fine. Spending lots of years thinking about it (on and off), a desire to work out any pain points, and having better tools available (like Phpstorm and Tracy) made it possible to finally massage out this pain point. Some work still remains to be done, but it’s mostly there and I’m feeling good about it. Stuff like this is key for the maintenance and longevity of the core, and involved a lot of time and effort, but makes very little difference to users, even if it makes a lot of difference to me in maintaining the core. It would make a boring blog post for sure—lots of work and changes, but no new toys to show for it. Nevertheless, it has that feeling of a good house cleaning, even if you can't see it from the outside. The scope of changes here means that there may actually be new bugs to work out, so to be on the safe side, consider 3.0.157 to be a little more “beta” than the dev branches usually are. Though I’m running it here on processwire.com and it’s working well. Beyond the fairly major updates to the Database classes, there are also a few new Sanitizer convenience methods that are primarily variations on existing ones, but useful ones for sure. Thanks for reading and have a great weekend!

This week I was back to focusing on the core and got quite a lot done. A lot of GitHub issue reports were resolved, plus several minor tweaks and additions were made in 3.0.156 as well. But the biggest update was the addition of the $pages->parents() API, which is something that I think you’ll likely have zero use for (and why I’m not putting it into a blog post) but something that the core itself will use quite a bit, and is a really nice improvement for the system and its scalability. So if you don’t mind some technical reading, read on. Whenever you call a $page->find() method ($page, not $pages) or use a “has_parent=“ in a selector, ProcessWire joins in a special table for the purpose called pages_parents. It uses this table to keep track of family relationships that aren’t otherwise apparent. For instance, let’s say we have page “g” that lives at path /a/b/c/d/e/f/g/. Page “g” only knows that it has page “f” as its parent. It doesn’t know that page “e” is its grandparent unless or until page “f” is loaded. Once “f” is loaded, then “f” can reveal its parent “e”. It works the same for every relationship down to the root parent “a”. So the pages are like a linked list or blockchain of sorts, where only 1 relationship forward or backward is known per page. The “pages_parents” table fills in this gap, enabling PW to quickly identify these relationships without having to load all the pages in the family. This is particularly useful in performing find() operations that you want to limit to a branch started by a particular parent. It’s the reason why we have both $pages->find() that searches the entire site, and $page->find() that limits the search within the branch started by $page. I haven’t paid much attention to the code behind this pages_parents table because it generally just worked, needing little attention. But I came across a couple of cases where the data in the table wasn’t fully accurate with the page tree, without a clear reason why. Then I became aware of one large scale case from a PW user where it was a huge bottleneck. It involved a large site (250k+ pages) and a recursive clone operation that appeared to involve hundreds of pages. But that operation was taking an unreasonable 10 minutes, so something wasn’t right. It came down to something going on with the pages_parents table. Once I dove into trying to figure out what was going on, I realized that if I was to have any chance of keeping track of it, we needed a dedicated API for managing these relationships and the table that keeps track of them. So that’s what got a lot of attention this week. While still testing here, it does appear initially that the 10 minute clone time has gone down to a few seconds, and everything about this relationship management is now rewritten, optimized and significantly improved. It was a lot of work, but absolutely worth it for PW. Rebuilding the entire table from scratch now takes between 2-3 seconds on a site with 250k pages and 150k relationships. The new API can be accessed from $pages->parents(). This API is really useful to the work that I do here (maintaining the core) but I’ll be honest, it’s probably not useful to most others, so I won’t go into the details here, other than to say I’m happy with it. But if you are interested, there are methods finding all the parents in a site, or a particular branch, and methods for rebuilding the pages_parents table, among others. Maybe more will be added to it later that actually would be useful in the public API, but for now I’ll likely leave it out of our public API docs. The $sanitizer->selectorValue() method also got a full rewrite this week (actually, one the last few weeks). It’s now quite a bit more comprehensive and configurable (see the new method options). The previous version was just fine, and actually still remains — you can use it by specifying [ ‘version’ => 1 ] in the $options argument to the selectorValue() method. But the new version is coded better and covers more edge cases, plus provides a lot more configurability for the times when you need it.

Beluga, I'm not much into carousels either, but also wouldn't claim there's no place for them. There's 1 small carousel on this entire site, and there was also one on the previous iteration of the site—the client has always liked it, and the customers react well to it. I really like this client for a lot of reasons, but one is that they are much more involved than most, know their technology, their product and their audience better than anyone I've worked with. The carousel is not my idea, but I trust and am certain the client knows their customers better than any self proclaimed experts online. I got a kick out of that linked anti-carousel site because it's a bit of a self own by whoever made it—it uses a carousel to make points that we likely would not have bothered to read if they weren't in a carousel. ?

Last week I told you guys about how I was working on development of a client’s site and deep diving into the details. Well this week was the same, except perhaps more so. Yesterday we finally launched phase one. There’s still more to do, and still working out some small bugs and such. This is a site I’ve been developing since back in the early versions of ProcessWire 2.x, which I think was nearly a decade ago. In fact, this was one of the first sites running ProcessWire 2.x, if I recall correctly. We’ve been keeping it up-to-date over the years, but this is the first time we’ve done a full front-end redo of the site, essentially starting from scratch on that, and spending a few months on it. But the admin side (fields, templates and page structure) remains roughly the same from where it was 10 years ago, and that’s what we’re going to redo in phase 2 (this year). There’s a lot of fields in this site, so I’m looking forward to really simplifying it with ProFields, repeaters and more — none of these existed when the original site was built. One thing really great about this iteration of the site is that it’s a ProcessWire community collaboration. Pete (our awesome forum administrator) did the design, as well as most of the front-end markup/css for the site. Jan (our awesome system administrator) setup the servers that it runs on, with AWS load balancer setup and everything related to that (he’s also one of the owners of the site). I did the other development stuff (the ProcessWire API side of things), making all the content fill all the markup in the right places, structure and organization, markup regions, search engines, optimization, etc., basically the stuff needed to get it all working in PW. This is the first time that I’ve developed a site where we can run new and old site side-by-side off the same ProcessWire installation. During development, the client could click a checkbox in their user profile and then they’d be using the new site. Behind the scenes, it does this using a fairly new feature in ProcessWire which is: $config->setLocation(‘templates’, ‘site/tpl’); So we had site/tpl/ having the new site templates, while site/templates/ had the old version. So one limitation for this phase 1 is that the old and new sites had to deliver the same exact content, even if the resulting output was very different. This site also uses custom page classes (another new feature), Markup Regions, and ProcessWire’s Functions API, and the Uikit 3 front-end framework. Pete also wrote a nice custom PW module for this site to handle localized weather forecasts — I ran out of time to get it in place yesterday, but that should be there next week hopefully. Yesterday we launched the site and we were finally able to start running it through its paces with live traffic. I thought I was going to be working on the PW core today, but you know that as soon as you launch a new site you always find things that need adjustment and can’t wait, so that was all of today. There’s more optimization work to do, and then there’s phase 2, where we start using ProFields and Repeaters to better isolate a lot of data and be able to implement the rest of Pete’s design in the parts that we weren’t able to in phase 1. This is where things like the current pricing tables (one example of a weak point) start to look really sharp. But I’m also looking forward to taking a little breather and catch up on some serious PW core work, issue reports and module updates over the next few weeks before diving into phase 2 of this site. I didn’t want to share the site quite yet because there’s still some details to take care of and such. But here it is Friday and I’ve got no other ProcessWire news to report, so was feeling a little guilty that I didn’t get more ProcessWire core work done over the last week, due to being focused on developing this site. But this was one of the first sites running ProcessWire, so it's an important one to me. And here it is about 10 years later, still the same installation (templates, fields, page tree) but with a brand new design and running the latest PW, and lots more to come. Tripsite.com

I hope that you all are having a good week and staying healthy. I’ve been very much focused on preparing a new ProcessWire-powered client’s website for launch, hopefully sometime next week. This is a site that’s already pretty well established with the search engines and such, so I’ve been going through all of the pre-launch QA in terms of optimization, SEO accessibility, mobile testing, getting the multi-language translators everything they need to translate, adding ProCache to the picture, and so on. For me this is one of the most fun parts of development, with relentless optimization and having it all come together… but also one of the most time intensive. This is where all the little details surface, things that need improvement become more apparent, and new ideas keep popping up. It’s also where lots of the bugs are discovered and fixed, and where I sometimes have to slap myself on the head for missing one thing or another, but then of course taking care of it. So that’s been most of this week, and I think likely at least half of next week too. But then it’s back to work on the core and modules, covering issue reports and more. I’ve actually been at work on some parts of the core and ProCache this week as well, but not to the point of finishing anything major. So you’ll see some updates on the core this week, and there’s also lots of stuff pending that’s not yet committed, and it’s all work in progress, so stuff to review later. With kids at home I’m still moving at a little slower pace than usual, but starting to get the hang of it, doing well and building momentum. While I don’t have much new to offer in this update, thanks for reading anyway, and I hope that you have a great weekend!

This post covers a few of the bigger updates in ProcessWire 3.0.154 and 3.0.155 on the dev branch. This includes a new function for live replacement of text in core and modules, a new method for creating canonical URLs, and some major upgrades to our $input->urlSegment() method that I think you’ll like! This continues from last week’s post in the ProcessWire support forum about 3.0.154 core updates and includes several new details— https://processwire.com/blog/posts/pw-3.0.155/

@Pete From this side, my wife's job is one that requires her to be on phone/video meetings most of the day, so she's got to lock herself away from us to focus on that, and the kids are mostly with me during the day. I'm good at focused attention, but not great at changing gears, so I'm still learning. My days used to be just coffee and code. Now things are much more diverse, here's a typical day: wakeup, breakfast, coffee, code, kids wakeup, resolve dispute, make breakfast, coffee, code, hear that TV is on, check in to make sure kids are doing school work, turn off TV, say “get back to school work”, try to figure out something with google classroom, worry about whether I'm doing enough to keep kids focused on school, code, get icloud request to approve download of some game on iPad, tell kids to “get back to work”, explain why several times, answer questions about school work, hear music from some game and decide to let it slide, 1 email, cycle power on router because wife says she can't connect to important video call, code, resolve kids dispute, put 1 kid in timeout, legos, clean up spill, 1 email, slack message, cook lunch, coffee, code, worry about whether kids are spending too much time on screens, phone call, reset wifi because daughter says Roblox won't connect, then work a little more, click the "snooze" button on a dozen emails so they return tomorrow, take kids outside to play while I work on house or yard, ride bikes, cook dinner, eat, netflix, cleanup dinner, go to bed, feel thankful to be healthy, sleep well. I'm definitely still adapting here, but I can't complain about the new normal because having everyone under one roof is comforting, especially at times like this. And while it's harder to get work done, it's mostly still getting done and I just feel thankful to be staying busy. Also, knowing there's nowhere to go but home, life is simpler in many ways.

Here’s a brief update on what’s new in the core since last week. Next week I’m going to compile a more complete overview of everything that’s in 3.0.154 (and more) and put it in a blog post. I wanted to do that today, but my kids have needed a lot of help with various things today (and this whole week actually) with all of this online school stuff and I’ve run out of time. We're a few weeks in and school is cancelled for the rest of the year due to the coronavirus (likely the same around the world). I'm still learning how to adapt to kids around all day and needing attention, but overall it's nice to have the family around more even if it is sometimes hard to get work done. So I’ll keep this sort of brief, and get into more details next week. Several updates were made to the core/LanguageFunctions.php (language functions) file, with biggest being the option to programmatically replace translation text (whether multi-language support is installed or not), useful for very easily updating core/module output to say exactly what you want, and without needing to install multi-language modules. No longer do you need to use various hooks to change text in one module or another. Also moved the __('text') configuration options to dedicated functions which makes it more clear to use, as well as much better for documentation purposes: wireLangEntityEncode(), wireLangTranslations(), wireLangReplacements(). You'll see these new functions in our docs by next week. More details and examples of these in action in next week’s blog post. The CKEditor version has been upgraded from 4.12.1 to 4.14.0. To see what’s changed, see the release notes for CKEditor 4.13.0, 4.13.1 and 4.14.0 here. There’s actually quite a lot here in terms of additions and fixes. Note especially the two security related fixes in 4.14.0 (though I think only the first may be applicable to us). While it looks like a rather unlikely combination of factors needs to come together, and with just the right person, it’s good to be aware of nevertheless. Of note is that these particular cases are entirely client side, before PW ever gets to see or filter the content, so this is something that only a CKEditor upgrade can fix. Updated the core Debug class to use PHP 7.3’s recommended hrtime (high resolution time) for debug timers, rather than microtome. This is only available in PHP 7.3 and newer. As far as I can tell it produces identical results to microtome (as far is our timers go) but the PHP documentation page indicates hrtime is what we should now be using, so that’s what we’re going with. I also did a little refactoring and minor improvements in the Debug class as well. Updated ProcessWire’s Inputfield rendering system so that you can customize markup by matching a field by name, and without hooks. This essentially takes part the Inputfield customization system developed for LoginRegisterPro and brings it into the core. This opens up a lot more customizability for those that want it. Various resolutions of GitHub issue reports, with more on the way. The above are just updates from the last week, but there’s plenty more in 3.0.154 as well. See ProcessWire Weekly #309 for details on Image and file field updates, new $database->columnExists() method and new Selectors::getOperators() method, and more. There are also the new expanded file and image properties, as discussed in #308 and the last blog post. Thanks for reading and I hope you all have a great weekend!

Last week I posted in the blog about current projects in progress, and consistent with that, this week I’ve been working primarily on the upgrades to our file/image fields. To start, this focuses on storing additional info in the DB table, including file size, created/modified user info, and for image fields: width and height. This is useful in making these properties searchable from page-finding selectors, but also a necessary pre-requisite to supporting external file systems. A “ratio” floating point property is now available on image fields as well (and also stored in the DB) and this value reflects the width divided by the height. This is a searchable property and will be useful for quickly finding portrait images, landscape images, square images, or any proportion you need to find. If you want to find images that are at least twice as wide as they are tall, then you’d be looking for a “ratio” of 2.0 or larger (“ratio>=2”). If you needed to find perfectly square images (i.e. 300x300) then you’d be looking for a ratio of 1.0. If you needed to find portrait images, then you’d be looking for images with a ratio under 1.0 (or twice as tall as wide would be 0.5). You can figure out what ratio you are looking for just by taking your target width and dividing by your target height. Longer term, combined with the other added properties, the goal is that this will help with finding pages that have images suitable for specific placements, proportions and dimensions. These new properties won’t be useful at first on existing sites because the data just isn’t present in the DB, even if the columns to store them are. They are populated over time whenever the page is saved. But I’m going to enable an option to have it populate all of this automatically whenever the field is even loaded. It’s actually there in the core right now, but I’ve got it disabled so I can test it out on a few more of my installations before others start using it. There were some other upgrades to the core as well, including improvements to the Selector and Selectors classes, addition of a $database->columnExists() method, lots of new (mostly internal) methods in FIeldtypeFile/FieldtypeImage, and some refactoring of the FieldtypeMulti class, among other minor updates. While there were a lot of changes and additions this week, it’s not stuff that you’ll likely be using right away, so I’m going to hold off on bumping the version till next week. Thanks for reading and have a great weekend!

This week we’ve got a few new and interesting core updates in progress, though none quite ready to release just yet. So rather than releasing version 3.0.154 today, I thought we'd instead take a brief look at what’s coming over the next few weeks… https://processwire.com/blog/posts/processwire-updates-and-additions-in-progress/