PW 3.0.163 – Core updates & ProCache v4

[ryan]

ProcessWire 3.0.163 adds a few new $pages hooks (see PW Weekly #323 for details), adds configurable module upload/install options (ProcessModule), and contains many other minor updates, code refactoring and optimizations. But by far, the majority of updates and commits are related to resolving more than a dozen recent issue reports. That will be the focus next week too, as the goal is to have the next master version out by the end of the month, or the first week of August. Priority focus is on any issues that might be bugs introduced between 3.0.148 (previous master) and 3.0.163, as we want to make sure at minimum we aren’t adding any new bugs from one master version to another. 

Regarding the new configurable module upload options, the intention here is to add additional safety by having the option of locking down the ability to install modules from the admin. As convenient as it is to be able to install and upgrade modules (during development) directly by URL, file upload or directory; the reality is that—depending on the case—it’s also not the safest thing to have on a client’s production site once development is finished. I think it’s best if module installation and upgrades are left to web developers, who are better equipped to resolve any technical issues that might arise during the process. Though it also depends on the installation, which is why I thought it should be configurable. So now you can specify which install options are available, and they can also depend on whether the site is in debug mode or not:

$config->moduleInstall = [
   // allow install from ProcessWire modules directory? 
  'directory' => true,

  // allow install by module file upload?
  'upload' => 'debug', 

  // allow install by download from URL?
  'download' => 'debug', 
]; 

Above are the current defaults, which can be changed by specifying your own preferred options in /site/config.php. Boolean true means always allowed, boolean false means never allowed, and string “debug” means: allowed if the site is in debug mode. (I’m currently debating on whether the ‘directory’ option should also be ‘debug’ as well.)  

In addition to these configuration options, the ProcessModule “New” tab now also provides instructions for manual installation of modules. None of us need it I know, but someone new to ProcessWire might see the prior “New” tab and not realize there’s a really simple and safe way to install modules from the file system. So the instructions just seemed to make sense there for consistency. 

ProCache 4.0 β released

Last week I mentioned a new version of ProCache would be coming out this week and version 4.0 of ProCache was released on Wednesday in beta form. It’s available for download now in the ProCache board download thread. This is one of the biggest upgrades for ProCache yet. If you are upgrading from a previous version, read the upgrade instructions in the README.txt file included with it, as it will guide you through some of the new features, and may save you from having to make an update to your .htaccess file. I mentioned much of this in last week's post, but here’s a summary of what’s new in this version of ProCache relative to the previous version:

• Major refactor of entire module.
• Now native to ProcessWire 3.x (ProcessWire 3.0.148+ recommended). 
• New .htaccess rules for static cache.
• New ability to select .htaccess version (v1 or v2, Tweaks tab).
• New option to specify how trailing slashes are handled (Tweaks tab).
• Upgrade SCSS compiler from version 0.7.8 to 1.1.1.
• Add Wikimedia LESS as additional option to Leafo LESS (select which you want on Tweaks tab).
• Improved per-template lifespan settings.
• Improved default and per-template behavior settings. 
• New cache clear behavior: Family (parent, siblings, children).
• New cache clear behavior: References (page that reference saved page).
• New cache clear override: No-self (skip clearing page that was saved).
• Per-template behaviors now supports clearing specific pages by ID or selector.
• Numerous minor fixes and optimizations throughout. 
• Removed direct .htaccess writing ability, replaced with live example file.
• Lots of new hookable methods for special cases. 
• New “Tests” tab with the following built-in tests: cache function and performance; cache clear behaviors tests; test http response headers; test for GZIP, LZW, zlib/deflate or Brotli compression; Test for keep-alive connection.

Thanks for reading and have a great weekend!

[Sam Howat]

I didn't know the configurable module upload functionality was coming and this was an awesome way to end the week. Can't wait to play with it!

[Zeka]

 

On 7/24/2020 at 11:10 PM, ryan said:

Lots of new hookable methods for special cases. 

@ryan Could you (or somebody who have access to the new version) please list names of these methods as I thinking about updating and buying a new licence. 

In my case, I need to be able to delete only specific cached urlSegment or page without its urlSegmetns independently from publishReady or saveReady hooks

[ryan]

@Zeka I don't think we've got hooks specific to what you are trying to do there, but I've pasted in the list of current hooks that ProCache provides for the static caching side of things. Can you expand on your need with an example? It may be something I can add, though want to make sure I fully understand what you are trying to do. 

// Is caching allowed for given page? Return true or false
// Hooks can modify return value to dictate custom caching rules
ProCache::allowCacheForPage(Page $page)

// Called when entire cache is being cleared
// Returns quantity of items cleared
ProCache::clearAll()

// Called right before a cache file is written
// Hooks can modify $content to adjust what is saved in cache file
ProCacheStatic::writeCacheFileReady(Page $page, $content, $file)

// Called right before a Page is about to have its cache cleared. 
// Hooks can make it return false to bypass clearing of page.
ProCacheStatic::clearPageReady(Page $page)

// Called to execute cache clearing behaviors for given Page
// Hooks can modify what behaviors are executed ($behaviors argument)
// Returns array indexed by behavior name each with a count of files cleared
ProCacheStaticBehaviors::executeCacheClearBehaviors(Page $page, array $behaviors)

 

[Zeka]

@ryan Thanks. 

I have two examples:

--------------

I have an RSS page which uses RSS template. 
This page outputs the main RSS feed, also there are several allowed URL segemtns for this template such: 
rss/google-news/
rss/yandex-news/
rss/ukrnet/
 etc. 
Each segment outputs platform-specific feed with different markup and pages that are listed in the feed (based on the selection from options field). 

So when any new post is published or saved cache for RSS page including its URL segments are cleared, but because new post could be listed only for Google News clearing cache for Yandex and Ukrnet segments is unnecessary. 

--------------

Next example 
I have sitemap page/template which outputs sitemaps index like: 

Spoiler

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://site.com.ua/ua/sitemap/categories/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/tags/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/pages/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2019-10/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2019-11/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2019-12/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2020-01/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2020-02/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2020-03/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2020-04/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2020-05/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2020-06/</loc>
</sitemap>
<sitemap>
<loc>https://site.com.ua/ua/sitemap/2020-07/</loc>
</sitemap>
</sitemapindex>

 

Based on URL segments I generate a sitemap for specific time period or template. The same thing here if new category or post is created I want to clear the cache only for relevant template URL segment or for segment that reprecent current month (there are several thousand posts per month so regenerating cache for each URL segment each time creates unnecessary load on the server). 

 

[ryan]

@Zeka Thanks for the examples. Reading what you write made me realize I've got a similar case too: https://www.tripsite.com/reviews/ — it's just one page, but hundreds (maybe thousands) of URLs underneath are all URL segment powered for 3 levels. What you are asking for would come in handy here for sure. In the hook example below (which is functional) I clear all /reviews/bike-tours/* segments on the /reviews/ page when a page using template "tour" is saved. And when a page using template "boat" is saved, it clears all of the /reviews/boat-[name]/ segments, where [name] is the name of the boat page that was saved. This is going to come in very handy here, and if I understand your case correctly, I think it's what you are looking for also?

$wire->addHook('ProCacheStaticClear::clearedPage', function($event) {

  $page = $event->arguments(0); // Page that was saved/cleared
  $clearSegment = '';

  if($page->template == 'tour') {
    // clear all /reviews/bike-tours* segments
    $clearSegment = 'bike-tours*';

  } else if($page->template == 'boat') {
    // clear all /reviews/boat-name/ segments
    $clearSegment = "boat-{$page->name}*";
  }

  if($clearSegment) {
    // clear segments below the /reviews/ page
    $reviews = $event->pages->get('/reviews/');
    $event->object->clearPage($reviews, [ 
      'urlSegmentStr' => $clearSegment
    ]);
  }
});

 

[Zeka]

@ryan 

Just to make sure that I understand you correctly when a page with 'tour' or 'bike' template is saved this hook clears cache only /reviews/bike-tours/* and  /reviews/boat-[name]/  segments.  Cache for review page and all other segments stay untouched.
If so, it looks like what I'm looking for. 

ProCacheStaticClear - is it something new in V4 or tripsite specific? 

Definitely it steps forward to save resources and more selective cache clearing process. But I would like to propose one more optimization relative to multilanguage setup. 

We also have two languages:  Ukrainian as default and Russian as secondary. Only 40% of news have a translation to the Russian language, but when a page is saved and doesn't have translation (secondary language is not active) ProCache clears the cache for all Russian pages and listings.
In our case, every news item page has tags, sections, categories, author reference fields (all these referenced pages also have pagination and language versions), could be listed in sitemaps, RSS feeds etc. So clearing cache for secondary language (probably hundreds (in some case thousand) of pages), when there are actually no changes for it could produce unnecessary load.  

I have found some example on tripsite.com where this optimization could be handy. 

https://www.tripsite.com/bike-boat/all/ - 106 Tours
https://www.tripsite.com/br/bicicleta-barco/all/ - 65 Tours
https://www.tripsite.com/es/bici-barco/all/ - 46 Tours

I can assume that not all tours available on every language and when a new relevant page with template 'tour' is saved cache for all these URLs (pages) and URL segments is cleared regardless of Portugues or Espanol languages are active for the saved tour.

What do you think? 

 

 

[ryan]

@Zeka

Quote

Just to make sure that I understand you correctly when a page with 'tour' or 'bike' template is saved this hook clears cache only /reviews/bike-tours/* and  /reviews/boat-[name]/  segments.  Cache for review page and all other segments stay untouched.

That's correct.

Quote

ProCacheStaticClear - is it something new in V4 or tripsite specific? 

V4 is split out into a bunch of different classes like this, as ProCache has grown quite a bit. 

Quote

Only 40% of news have a translation to the Russian language, but when a page is saved and doesn't have translation (secondary language is not active) ProCache clears the cache for all Russian pages and listings.

It's possible I might not understand what you mean. But ProCache only clears cache files that exist. It knows ahead of time which ones exist because there's a corresponding DB table entry for every cache file. So it shouldn't attempt to find or clear files for any languages that don't have the content available in the cache. Meaning, so long as the page isn't published in the language that it lacks content in, ProCache isn't going to be attempting to write or clear cache files for that page in that language it's not available in (i.e. no extra overhead). 

But if what you are looking for is more logic in determining whether a page is cleared for a particular language, that clearPage() call in the earlier hook I mentioned does also accept a 'language' option which limits the clear to a particular language. You can also tell it to clear only a particular URL segment, or segments matching a wildcard pattern. Here's the options for the clearPage method: 

   /**
     * Clear the cache for a specific page 
     *
     * Default behavior is to clear for all languages, paginations and URL segment variations.  
     * To clear only specific languages, paginations or URL segments, use the options. 
     *
     * @param Page $page
     * @param array $options
     *  - `language` (string|int|Language): Clear only this language (default='')
     *  - `urlSegmentStr` (string): Clear only entries matching this URL segment string, wildcards OR regex OK (default='')
     *  - `urlSegments` (array): Clear only entries having any of these URL segments (default=[])
     *  - `pageNum` (int|bool): Clear only pagination number (i.e. 2), true to clear all paginations, false to clear no paginations (default=0)
     *  - `clearRoot` (bool|null): Clear root index of page path? (default=false when URL segments or paginations requested, true otherwise)
     *  - `rmdir` (bool): Remove directories rather than index files? (default=false)
     *  - `getFiles` (bool): Get array of files that were cleared, rather than a count? (default=false)
     * @return int|array Quantity or array of files and/or directories that were removed
     *
     */

Quote

I can assume that not all tours available on every language and when a new relevant page with template 'tour' is saved cache for all these URLs (pages) and URL segments is cleared regardless of Portugues or Espanol languages are active for the saved tour.

I think we've got all the tools we need to support that kind of logic on the hook side, but ProCache doesn't do that automatically at present. It's a good idea though, I can explore it further.