Jump to content


  • Content Count

  • Joined

  • Last visited

  • Days Won


ryan last won the day on August 12

ryan had the most liked content!

Community Reputation

17,983 Excellent

About ryan

  • Rank

Contact Methods

  • Website URL

Profile Information

  • Gender
  • Location
    Atlanta, GA

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. This week we’re proud to announce the newest ProcessWire master version 3.0.164. Relative to the previous master version (3.0.148) this version adds a ton of new and useful features and fixes more than 85 issues, with more than 225 commits over a period of 7 months. There were also a lot of technical changes in the core, including some pretty significant refactoring of several classes to further optimize and improve upon our already solid core, making it that much better. I’m confident this master version is one of our best and if you are running an older version I’d encourage you to upgrade when convenient to do so—I think you will really like what this version brings! Read all about what’s new in the latest blog post: https://processwire.com/blog/posts/pw-3.0.164/
  2. @schwarzdesign I was never able to duplicate that one. Pasting in old password makes the new password editable when I test here. But I think this one has come up before and it had something to do with some other admin enhancement module, but I don't remember for sure. There's a reply in that thread that says it only happens of entity encoding textformatter is disabled for the title field. I think it's rare for someone to disable that textformatter on the title field and something I wouldn't usually recommend doing. But I think we can also accommodate the request in that report, just wanted to return to it later when back on the dev branch because it's a change of behavior that I think needs more thorough dev branch testing.
  3. In preparation for the next master version, this week no new features were added, but just like last week, more than a dozen issue reports were resolved. Having focused largely on fixing various issues over the last month, I feel pretty confident that the current dev branch is significantly more solid than the 3.0.148 master version. It adds and improves a whole lot, and also fixes a lot. And to the best of my knowledge, there aren’t any new issues between 3.0.146 and 3.0.164 that haven’t already been fixed. Basically, I don’t think it makes any sense to keep all these updates exclusive to the dev branch any longer, so have merged it to master, today. Consider it a soft launch, as I haven’t made it an official tagged version yet. Maybe I’m shy, but wanted to wait till Monday before Git tagging it and making it official. The master branch has a different audience than the dev branch, and so there’s always that possibility that some new issue will appear that hasn’t on the dev branch, and wouldn’t have. So we’ll let it marinate on the master branch for the weekend before broadcasting it very far. By this time next week, I should have a blog post ready that covers all that’s new in this version, which is 226 commits ahead of the previous master (3.0.148), so there’s a lot to cover. I want to thank you all that have been helping to identify and report issues on GitHub as they come up. Having covered a lot of issue reports over the last month, I can see a lot of effort goes into preparing many of the reports. Your work is appreciated. This month I focused primarily on the reports that I thought were likely to make the most difference to the most people. I also focused on issues that I thought could be accommodated without introducing potentially new issues or new code to test. Of course, not every report could be covered, and there’s always more to do, so I’ll be getting back to it on the dev branch here soon. In addition, I’ve held off on some new things I’ve wanted to add for awhile (a feature freeze of sorts) in preparation for an end-of-month master version. I’m looking forward to outlining all that’s new in next week’s blog post. Until then, thanks for reading and if you get a chance to test out the new 3.0.164 version, please do and let me know how it goes (both master and dev branches are identical right now). I hope you have a great weekend!
  4. @Zeka That's correct. V4 is split out into a bunch of different classes like this, as ProCache has grown quite a bit. It's possible I might not understand what you mean. But ProCache only clears cache files that exist. It knows ahead of time which ones exist because there's a corresponding DB table entry for every cache file. So it shouldn't attempt to find or clear files for any languages that don't have the content available in the cache. Meaning, so long as the page isn't published in the language that it lacks content in, ProCache isn't going to be attempting to write or clear cache files for that page in that language it's not available in (i.e. no extra overhead). But if what you are looking for is more logic in determining whether a page is cleared for a particular language, that clearPage() call in the earlier hook I mentioned does also accept a 'language' option which limits the clear to a particular language. You can also tell it to clear only a particular URL segment, or segments matching a wildcard pattern. Here's the options for the clearPage method: /** * Clear the cache for a specific page * * Default behavior is to clear for all languages, paginations and URL segment variations. * To clear only specific languages, paginations or URL segments, use the options. * * @param Page $page * @param array $options * - `language` (string|int|Language): Clear only this language (default='') * - `urlSegmentStr` (string): Clear only entries matching this URL segment string, wildcards OR regex OK (default='') * - `urlSegments` (array): Clear only entries having any of these URL segments (default=[]) * - `pageNum` (int|bool): Clear only pagination number (i.e. 2), true to clear all paginations, false to clear no paginations (default=0) * - `clearRoot` (bool|null): Clear root index of page path? (default=false when URL segments or paginations requested, true otherwise) * - `rmdir` (bool): Remove directories rather than index files? (default=false) * - `getFiles` (bool): Get array of files that were cleared, rather than a count? (default=false) * @return int|array Quantity or array of files and/or directories that were removed * */ I think we've got all the tools we need to support that kind of logic on the hook side, but ProCache doesn't do that automatically at present. It's a good idea though, I can explore it further.
  5. @Zeka Thanks for the examples. Reading what you write made me realize I've got a similar case too: https://www.tripsite.com/reviews/ — it's just one page, but hundreds (maybe thousands) of URLs underneath are all URL segment powered for 3 levels. What you are asking for would come in handy here for sure. In the hook example below (which is functional) I clear all /reviews/bike-tours/* segments on the /reviews/ page when a page using template "tour" is saved. And when a page using template "boat" is saved, it clears all of the /reviews/boat-[name]/ segments, where [name] is the name of the boat page that was saved. This is going to come in very handy here, and if I understand your case correctly, I think it's what you are looking for also? $wire->addHook('ProCacheStaticClear::clearedPage', function($event) { $page = $event->arguments(0); // Page that was saved/cleared $clearSegment = ''; if($page->template == 'tour') { // clear all /reviews/bike-tours* segments $clearSegment = 'bike-tours*'; } else if($page->template == 'boat') { // clear all /reviews/boat-name/ segments $clearSegment = "boat-{$page->name}*"; } if($clearSegment) { // clear segments below the /reviews/ page $reviews = $event->pages->get('/reviews/'); $event->object->clearPage($reviews, [ 'urlSegmentStr' => $clearSegment ]); } });
  6. @Zeka I don't think we've got hooks specific to what you are trying to do there, but I've pasted in the list of current hooks that ProCache provides for the static caching side of things. Can you expand on your need with an example? It may be something I can add, though want to make sure I fully understand what you are trying to do. // Is caching allowed for given page? Return true or false // Hooks can modify return value to dictate custom caching rules ProCache::allowCacheForPage(Page $page) // Called when entire cache is being cleared // Returns quantity of items cleared ProCache::clearAll() // Called right before a cache file is written // Hooks can modify $content to adjust what is saved in cache file ProCacheStatic::writeCacheFileReady(Page $page, $content, $file) // Called right before a Page is about to have its cache cleared. // Hooks can make it return false to bypass clearing of page. ProCacheStatic::clearPageReady(Page $page) // Called to execute cache clearing behaviors for given Page // Hooks can modify what behaviors are executed ($behaviors argument) // Returns array indexed by behavior name each with a count of files cleared ProCacheStaticBehaviors::executeCacheClearBehaviors(Page $page, array $behaviors)
  7. @JeevanisM Thanks, I have fixed the table salt comment error. The version in the button is on a cache, so it'll update eventually.
  8. ProcessWire 3.0.163 adds a few new $pages hooks (see PW Weekly #323 for details), adds configurable module upload/install options (ProcessModule), and contains many other minor updates, code refactoring and optimizations. But by far, the majority of updates and commits are related to resolving more than a dozen recent issue reports. That will be the focus next week too, as the goal is to have the next master version out by the end of the month, or the first week of August. Priority focus is on any issues that might be bugs introduced between 3.0.148 (previous master) and 3.0.163, as we want to make sure at minimum we aren’t adding any new bugs from one master version to another. Regarding the new configurable module upload options, the intention here is to add additional safety by having the option of locking down the ability to install modules from the admin. As convenient as it is to be able to install and upgrade modules (during development) directly by URL, file upload or directory; the reality is that—depending on the case—it’s also not the safest thing to have on a client’s production site once development is finished. I think it’s best if module installation and upgrades are left to web developers, who are better equipped to resolve any technical issues that might arise during the process. Though it also depends on the installation, which is why I thought it should be configurable. So now you can specify which install options are available, and they can also depend on whether the site is in debug mode or not: $config->moduleInstall = [ // allow install from ProcessWire modules directory? 'directory' => true, // allow install by module file upload? 'upload' => 'debug', // allow install by download from URL? 'download' => 'debug', ]; Above are the current defaults, which can be changed by specifying your own preferred options in /site/config.php. Boolean true means always allowed, boolean false means never allowed, and string “debug” means: allowed if the site is in debug mode. (I’m currently debating on whether the ‘directory’ option should also be ‘debug’ as well.) In addition to these configuration options, the ProcessModule “New” tab now also provides instructions for manual installation of modules. None of us need it I know, but someone new to ProcessWire might see the prior “New” tab and not realize there’s a really simple and safe way to install modules from the file system. So the instructions just seemed to make sense there for consistency. ProCache 4.0 β released Last week I mentioned a new version of ProCache would be coming out this week and version 4.0 of ProCache was released on Wednesday in beta form. It’s available for download now in the ProCache board download thread. This is one of the biggest upgrades for ProCache yet. If you are upgrading from a previous version, read the upgrade instructions in the README.txt file included with it, as it will guide you through some of the new features, and may save you from having to make an update to your .htaccess file. I mentioned much of this in last week's post, but here’s a summary of what’s new in this version of ProCache relative to the previous version: Major refactor of entire module. Now native to ProcessWire 3.x (ProcessWire 3.0.148+ recommended). New .htaccess rules for static cache. New ability to select .htaccess version (v1 or v2, Tweaks tab). New option to specify how trailing slashes are handled (Tweaks tab). Upgrade SCSS compiler from version 0.7.8 to 1.1.1. Add Wikimedia LESS as additional option to Leafo LESS (select which you want on Tweaks tab). Improved per-template lifespan settings. Improved default and per-template behavior settings. New cache clear behavior: Family (parent, siblings, children). New cache clear behavior: References (page that reference saved page). New cache clear override: No-self (skip clearing page that was saved). Per-template behaviors now supports clearing specific pages by ID or selector. Numerous minor fixes and optimizations throughout. Removed direct .htaccess writing ability, replaced with live example file. Lots of new hookable methods for special cases. New “Tests” tab with the following built-in tests: cache function and performance; cache clear behaviors tests; test http response headers; test for GZIP, LZW, zlib/deflate or Brotli compression; Test for keep-alive connection. Thanks for reading and have a great weekend!
  9. Greetings from the sunny covid hotspot state of Georgia, where we haven’t left the house since March. And now getting ready for the kids to start a new school year from home with virtual learning. Everyone delivers everything now, so there’s no need to go out to a grocery store anymore (or go anywhere). I live about a mile from the CDC, so our school district has more kids with parents working at the CDC than any other. That gives me some comfort, knowing that I won’t be sending my kids back to school until the experts at the CDC are willing to; when it’s really and truly safe. Though I don’t think it’s going to be safe for a long, long time. The US is a rudderless ship right now, so we just have to ride it out. Thankfully, we’re all staying safe and keeping busy. The kids are building houses in Roblox (an online game addiction they have), we’ve converted our yard to be a summer camp, and converted the basement to be a gym, while we clear more space to start building out a massive N-scale train set—my 3 locomotives still work perfectly, even after 35 years of storage. And I’ve been learning how to manage chlorine and PH in an inflatable kids pool that keeps the family cool in the hot weather. The kids miss school and other activities, my wife misses being at her office and people she works with, and we all miss our friends and family, but it’s the way things are now, and I’m just grateful to have my immediate family home and safe; and in place where we can ride out the storm. I’m also really glad that I can work on the ProcessWire core and modules for pretty much the entire work day, and enjoying coding as much as I ever have; feeling great about where ProcessWire is and where it’s going, thanks to all of you. I’ve been working on the latest ProCache version the entire week, so not many core updates to report today other than some new hooks added to the Pages class (they are hooks that the new ProCache can use as well). I’d hoped to have this version of ProCache finished by now, but I keep finding more stuff to improve, so decided give it another 2 days of work and testing, and if all looks good, it’ll be ready to release, which will be next week. This version is essentially a major refactor, where just about every line of code has been revisited in some form or another. But if you are already a ProCache user, you’ll also find it very familiar. While I don’t have it posted for download today, below is a brief look at what’s new. Completely new .htaccess rules (v2) that take up a lot less space, especially when using multiple hosts, schemes or extensions. Ability to choose .htaccess version (v1 or v2). ProCache now creates an example .htaccess-procache file that you can rename and use or copy/paste from. ProCache now has a built-in URL testing tool where you can compare the non-cached vs. cached render times. New setting to specify how ProCache delivered URLs should respond to trailing vs. non-trailing slashes in URL. Significant refactor that separates all ProCache functions into separate dedicated classes. Improved custom lifespan settings with predefined template lines. Improved behavior settings with predefined template lines and simpler letter (rather than number) based definitions. Ability to specify predefined cache clearing behaviors, specific pages to clear, or page matching selectors, from within the ProCache admin tool. New predefined cache clearing behavior: Reset cache for family of saved page (parents, siblings, children, grandchildren, and all within). New predefined cache clearing behavior: Reset cache for pages that reference saved page (via Page references). New versions of SCSS and LESS compilers. ProCache is completely ProcessWire 3.x native now (previous versions still supported PW 2.x even if 3.x was recommended). Numerous other improvements, fixes and optimizations throughout. I’ve previously mentioned a built-in crawler in ProCache. That part has been moved to a separate module called ProCacheCrawler and will be released a little later in the ProCache board. It was taking a little too much time to develop, so I didn’t want to hold up the rest of ProCache while I developed that. When installed, ProCache communicates with the crawler, identifying and adding URLs to a queue to be crawled and primed for the cache. What it does is pretty cool already, but it needs more time to develop. It’s also something that depends on being run regularly at intervals (like with CRON) so it’s a little bit of a different setup process than the rest of ProCache, which is another reason why I thought I’d develop is as a separate module. I’ll be working more on finishing development of the crawler later in the year, after the next master version of ProcessWire core is released. Next week I'll have the new ProCache version ready for download as well as a new core version on the development branch. It will focus mostly on fixes for issue reports as we continue working towards the next master version. Thanks for reading and have a great weekend!
  10. Relative to ProcessWire 3.0.161, version 3.0.162 contains 24 commits that continue upgrades/improvements to selector operators, fix various minor issues, add new API convenience methods, improve documentation, optimize and refactor various portions of code and DB queries, and much more. For full details, see the dev branch commit log as well as last week’s post. Next week I hope to finally finish up a new version of ProCache and continue with some additional core to-do items. By early August my hope is that we’ll have the next master branch version ready. Also added this week is a new dedicated documentation page on this site that covers all of ProcessWire’s selector operators, including all the newly added ones here: selector operators. Thanks for reading and have a great weekend!
  11. Good article and postgresql looks interesting with its search capabilities, thanks. Though none of these really solve what I was after here. I experimented quite as bit with stemming and different stemming libraries. Though they all did roughly the same thing. When it came to searching, stemming just wasn’t that useful. WireWordTools originally had a stemming library and methods, and the appropriate fulltext queries included the word stems with wildcards. In the end, it just wasn’t helpful most of the time. And in the few cases where it was worthwhile, it was redundant, though far less thorough, than what we already had with inflection and lemmatisation. So while stemming can have its uses, it’s not even half way there, if trying to build a smart search. Cool nevertheless that they have it built-in apparently. As far as accent support, ranking and fuzzy search, these are all things that MySQL does as well, though maybe there are differences in how they do them. For instance, MySQL supports “sounds like” and also supports pluggable parsers for fulltext searches. Fuzzy search also isn't what I'm after here, but certainly interested in exploring in the future. For me the most useful thing by far is boolean mode searches, particularly in InnoDB, which has a full-text engine modeled on Sphinx. Boolean mode searches are really very powerful, enabling you to specify what’s required, what’s excluded, matching of words or phrases, partial matching of words with wildcards, specifying noise words, isolating distance between words, adjusting ranking up or down on a per-word basis, grouped expressions and nested subexpressions. All while being incredibly fast. I’m pretty thrilled with what MySQL supports here and what it brings to ProcessWire. Postgresql looks very nice too, but for our needs, I don’t feel we are lacking anything relative to it. I think anyone that would say that as a general thing is not very familiar with what MySQL fulltext supports, or maybe is thinking of fulltext support where it was back a long time ago. For ProcessWire and the scale that most use it at, MySQL fulltext is really a sweet spot, enabling PW to deliver enormous power and capability when it comes to search features.
  12. @bernhard I didn't come up with the dictionary words in the JSON files, they are converted from an existing one (here) and apparently the original source is wordnet.princeton.edu. So I'm not sure if those particular words are intended or mistakes. New to me, but "wa" and "wo" are actual English words. Though as far as I can tell they aren't related to "was" or "will". I can't imagine those two instances will ever be helpful for our intended use case so maybe it makes sense to remove them. My plan was to keep looking for more existing dictionaries and continue to merge them into the one in WireWordTools so that it becomes more comprehensive over time.
  13. This week I'm not bumping the version number just yet because I've got lots of work in progress. The biggest thing so far is something I hinted at last week. Basically, I like what the addition of the MySQL query expansion operators have brought (per posts last week and week before), but they also reveal what's lacking: something as simple as a search for "books" still can't directly match the word "book". But that's the most basic example. It's not a limitation of ProcessWire, but just the type of database indexes in general. I think it'd be amazing if ProcessWire had the ability of being really smart about this stuff, able to interpolate not just plurals vs. singulars but related words. In a perfect world, this is what query expansion would do (in addition to what it already does). But the reality is that it involves all kinds of complicated logic, rules and dictionaries; well beyond the scope of even a database. And it can be vastly different depending on the language. So this isn't something we can just add to the core and have it work. On the other hand, I figured maybe we should just put in a hookable method that just pretends the ability was there. Then people could hook it and make it respond with variations of words, according to their needs. The searches that use query expansion could then call this method and use whatever it returns... for when someday the ability is there. So I went ahead and added that hook — WireTextTools::wordAlternates(). And our database-searching class (DatabaseQuerySelectFulltext) now calls upon it, just in case an implementation is available. Well, after getting that hook added and having our class call it, naturally I wanted to test it out. So I got to work on it and came up with this module: WireWordTools. The WireWordTools module provides an API for English word inflection and lemmatisation. And it hooks that new method mentioned above, so that you can install it and immediately have it bring your searches to the next level. While it only helps for English-language searches, maybe we'll be able to add more languages to it, or maybe it'll lead to other modules that do the same thing for other languages. The expanded/alternate words are only used for searches that use the new query expansion operators, which are the ones that have a "+" in them: ~+=, ~|+=, *+=, **+=. They all can return similar results, but are weighted differently. Unlike most operators, where the logic is direct and you can expect them to always behave the same way, these query expansion operators are more subjective, and ones I think we should intend to keep tweaking and improving over time to continually improve the quality of the results they return. Basically, they are geared towards building site search engines, so I think it makes sense for us to pursue anything that makes them better at that, rather than aiming to always have them return the same thing. I am currently testing out the ~|+= operator ("contains any words expand") on our main site search engine here, along with the WireWordTools module. Finally, searching for "books" does match "book" too, and a lot more. More to be done here, but it's a good start hopefully.
  14. Version 3.0.161 on the dev branch continues with the updates optimizing our support for the new selector operators introduced in last week's blog post for 3.0.160. Last week I was still kind of figuring it out and the code still needed some refactoring and optimization. This week several parts have been rewritten and it's been improved quite a bit. Though the end result is still very similar to what was demonstrated last week, but now it's a lot more performant and solid. One thing new this week that's also kind of fun: you can now use more than one operator in any "field=value" selector expression, at least from the API side. And it works anywhere that you might use a selector, whether querying the database or something in memory. I think the best way to explain it is with an example. Let's say that you want to find all pages with a title containing the phrase "hello world" — you'd do this using the "contains text/phrase" operator: *= $pages->find("title*=hello world"); But let's also say that if you don't find any matches, you want to fallback to find any pages that contain the words "hello" and "world" anywhere in the title, in any order. We'd use the "contains all words" operator "~=" to do that. So now you can add that operator to the existing one, and it'll fallback to it if the first operator fails to match. So we'll append the "contains all words" operator to the previous one: ~= $pages->find("title*=~=hello world"); Cool huh? But maybe we still aren't finding any matches, so we want to fallback to something even broader. So if the phrase match fails, and the words match fails, now we want to fallback to find any pages that contain the world "hello" OR "world", rather than requiring them both. For that we can use our new "contains any words" operator: ~|= $pages->find("title*=~=~|=hello world"); This example is getting a bit contrived now, but let's say that if we still haven't found a match, we want it to find any pages that have any words starting with "hello" or "world", so it would find pages with words like "helloes", "worlds", "worldwide", etc. That's a job for our new "contains any partial words" operator: ~|*= $pages->find("title*=~=~|=~|*=hello world"); Okay last one I promise—you probably wouldn't stack this many in real life, but stay with me here. Let's say the query still didn't find anything, and as a final fallback, we want it to find any words LIKE "hello" or "world", so that those terms can match anywhere in words, enabling us to find pages with words like in the previous example, but also words like "phellogen", "othello", "underworld", "otherworldly", etc., and that's a job for our new "contains any words like" operator: ~|%= $pages->find("title*=~=~|=~|*=~|%=hello world"); So that looks like a pretty complex operator there, but as you've seen by following the example, it's just these 5 appended operators to each other: *= Contains phrase ~= Contains all whole words ~|= Contains any whole words ~|*= Contains any partial words ~|%= Contains any words like I think a more likely scenario in a site search is that you might stack two operators, such as the *= followed by the ~|*=, or whatever combination suits your need. By the way, you can do this with any operators, not just the text searching ones. But if you didn't read the blog post last week, also be sure to check out the other new operators in addition to those above: *+=, **=, **+=, ~*=, ~+=, ~~=, ~%=, #=. I think these new operators help out quite a bit with ProcessWire's text searching abilities. But there's one thing that's kind of a common need in search engines that's not easily solved, and that's the handling of singular vs. plural. At least, that's a common issue when it comes to English (I'm assuming so for other languages, but not positive?) MySQL fulltext indexes can't differentiate between singular and plural versions of words, so they index and match them independently as completely different words. This can be unexpected as clients might type in "goose" and expect it's also going to match pages with "geese". I've already got something in the works for this, so stay tuned. Thanks for reading and hope you have a great weekend!
  15. @Robin S That's correct that the combination is not possible for technical reasons. The query expansion is a feature of MySQL fulltext indexes in a standard natural language match/against query, so it's not something that can be used with LIKE (which does not use fulltext index), and also can't be used with match/against boolean mode (which is how partial word matches can use fulltext). So there are only certain operators we can use query expansion with. My experience so far is that query expansion is pretty inconsistent as to when it's useful. But since it's something that MySQL supports I thought PW should provide the option. If a search returns no results, query expansion isn't going to help, because it needs there to be matches in order to expand upon them. I'm currently thinking I might use it on one or two word searches that might otherwise only return a small number of results. If your goal is to bring in as many results as possible I would start experimenting with the "Contains match and expand" **+= operator, which uses the standard MySQL match/against logic, and doesn't require all search words to be present. Though depending on the search, once you go past one or two words with query expansion, it tends to add noise to the results. So I've been thinking a few of these new operators might be made configurable to automatically substitute another operator under certain conditions, like to a non-expand version depending on word count, or automatically substitute another operator when one produces no results.
  • Create New...