PW 3.0.218 – Core updates

[ryan]

Continuing from last week's post and discussion, ProcessWire 3.0.218 decouples the modules system from the cache system. Now the modules system maintains its own internal caches (at least once you do a Modules > Refresh). It'll still use the $cache API as a backup (temporarily), but now you can safely export the database without the "caches" table, or even delete the "caches" table, if you want to. It'll get re-created as needed. 

In this version, work also continued on the new WireCacheInterface (and major updates in WireCache) so that we could support external modules to handle cache storage. This capability is kind of similar to how we support 3rd party WireMail and WireSessionHandler modules. The first example is WireCacheDatabase, which is the default cache storage handler for the core.

And today we have a new module called WireCacheFilesystem that replaces the default WireCache database storage with a file-system based storage, once installed.

It's not yet clear if there are major benefits one way or the other (cache in database vs. file system), as I've not been able to put all this new code through performance testing yet. I'd definitely be interested to hear if anyone has a chance to test things out. I expect the file system might be faster for reading caches, while the database may be faster for writing caches. At least that's what I found with a few preliminary experiments, but they haven't been very thorough, so take that with a grain of salt.

I thought we needed at least 2 examples of classes implementing WireCacheInterface before we'd be ready to support potential 3rd party WireCache modules. I imagine that 3rd party modules getting into dedicated cache options independent of database or file system is where we'll start to see major performance benefits. At least for sites that use the cache heavily. That's all for this week, have a great weekend! 

[Ivan Gretsky]

Ahh! Reading this in the early morning laying in bed. Almost always start the weekend with PW blog. What I read makes me smile. A great weekend indeed! Thanks!

[BitPoet]

11 hours ago, ryan said:

I imagine that 3rd party modules getting into dedicated cache options independent of database or file system is where we'll start to see major performance benefits.

About that: I'm itching (as far as time allows) to turn CacheRedis into a WireCacheInterface compatible module, but there might be a few stumbling blocks. I see that your modules support searching by expiry, which is not something most key-value-stores are designed for since they take care of expiry on their own. It's okay and possible to read the expiration value for an individual key, though, so I'm not completely sure yet how much of an impact a lack of that is going to have, and which applications besides maintenance are affected. More of a headache will be handling expiration constants like WireCache::expireNever, WireCache::expireReserved etc. since they hold dates in the past, meaning entries with those would expire the moment they are written. My gut feeling tells me that a lot more refactoring in WireCache will be necessary to decouple most of the expiration logic from the cacher implementation, but it might be possible to handle most cases by intercepting those values and applying a different logic. Not sure if I'll find time this (long, here in Germany) weekend to take a closer look.

Have a great weekend as well!

[ryan]

@BitPoet

Quote

I see that your modules support searching by expiry, which is not something most key-value-stores are designed for since they take care of expiry on their own.

This is for the maintenance processes. WireCache performs maintenance every 10 mins and whenever any page or template is saved. The page/template save maintenance only happens if rows have expiry dates prior to WireCache::expireNever. Finding by expiry dates can be optional if the cache (Redis) handles expiration on its own.

Quote

More of a headache will be handling expiration constants like WireCache::expireNever, WireCache::expireReserved etc. since they hold dates in the past, meaning entries with those would expire the moment they are written. 

I'm also not sure how much need for these two constants anymore since the modules are now decoupled from the cache. Though I think TracyDebugger uses expireNever. You can just replace WireCache::expireNever with some date far in the future (a year?). 

The WireCache::expireReserved can be treated the same as expireNever. It was primarily for the modules-system and I will likely deprecate this option since it's no longer needed. Its only purpose is to behave like expireNever but also prevent the rows from being deleted when a "delete all" action is being performed. We don't need that anymore.

There are a few expirations prior to expireNever that could also considered. Either that, or the WireCacheInterface module could just indicate it doesn't support them. Since these are all dates before WireCache::expireNever, they can be easily filtered in the same way. If the cache can't search by expiry, then implementing them would mean mapping to some other property that is searchable by the cache handler. Any WireCacheInterface module doesn't need to know about any of these since all the logic for finding them is mapped to expiration dates: 

• WireCache::expireSave - This expiration means that the cache row should be deleted whenever any Page or Template is saved. 
   
• WireCache::expireSelector - This is like expireSave above except that it indicates the cache value is JSON-encoded along with a property named 'selector' and this contains a page-finding selector. WireCache loads them all and when the saved page matches any selector then the cache row is deleted. 
   
• WireCache maps template IDs to dates by doing $expire=date('Y-m-d H:i:s', $template->id); and that can be reversed with $templateID=strtotime($expire); When a page having that template ID is saved, the cache row having that templateID-based expire date is deleted. Like with the above, the WireCacheInterface module doesn't need to know anything about this other than finding rows having those expiration dates, and presumably those could just be mapped to something else. 

I'm not sure it's necessary to support these other than mapping expireNever to a future date, as I think very few actually use these features. 

Quote

My gut feeling tells me that a lot more refactoring in WireCache will be necessary to decouple most of the expiration logic from the cacher implementation,

From what you've described, Redis handles a lot of the stuff that WireCache does on its own. If the cache handler (Redis) can handle these tasks on its own, that's great. If I'm understanding what you've described correctly, here's how I would handle it:

• When find($options) receives a request containing $options['names'] to find, then ignore anything in the $options['expires'], since Redis is never going to return an expired row.
   
• When find($options) receives a request with empty $options['names'], and a populated $options['expires'], just return a blank array (no results). 
   
• When save() receives an $expire date of expireNever or expireReserved, substitute some far future date instead. 
   
• When save() receives an $expire date prior to expireReserved, throw an Exception to say the feature isn't supported, or substitute 1 hour, 1 day, or whatever you think is appropriate.

Once it works with those, if you wanted to support the full feature set, then you could always go back and see about mapping expiration dates less than or equal to WireCache::expireNever to some other searchable property (and using a far future expiration date). But very few people actually use the page/template/selector clearing features (as far as I know), and the core doesn't need them, so they could be considered optional.

[adrian]

6 hours ago, ryan said:

Though I think TracyDebugger uses expireNever.

Let me know if you need those changed.
https://github.com/search?q=repo%3Aadrianbj%2FTracyDebugger expireNever&type=code

[ryan]

@adrian I don't think there's any need to change it, as cache handlers that don't support past dates can substitute a future date, i.e. 

if($expires === WireCache::expireNever) {
  $expires = wireDate(WireCache::dateFormat, '+1 YEAR'); 
}

 

[MarkE]

On 5/27/2023 at 3:22 PM, ryan said:

I'm not sure it's necessary to support these other than mapping expireNever to a future date, as I think very few actually use these features. 

I've not been following this very closely, but this statement caught my eye and caused me to worry. Are you saying that (for example) the selector string option for $cache->save (as per https://processwire.com/api/ref/wire-cache/save/) will no longer be supported? That would be a big problem for me as one of my web apps uses it extensively. The app provides a spreadsheet-like user interface but only updates data if precedent data has been changed (otherwise every 'cell' gets recalculated on every page view which would be a huge delay). Maybe I misunderstand, but I would be grateful for clarification.

[BitPoet]

On 5/27/2023 at 4:22 PM, ryan said:

From what you've described, Redis handles a lot of the stuff that WireCache does on its own. If the cache handler (Redis) can handle these tasks on its own, that's great. If I'm understanding what you've described correctly, here's how I would handle it:

• When find($options) receives a request containing $options['names'] to find, then ignore anything in the $options['expires'], since Redis is never going to return an expired row.
   
• When find($options) receives a request with empty $options['names'], and a populated $options['expires'], just return a blank array (no results). 
   
• When save() receives an $expire date of expireNever or expireReserved, substitute some far future date instead. 
   
• When save() receives an $expire date prior to expireReserved, throw an Exception to say the feature isn't supported, or substitute 1 hour, 1 day, or whatever you think is appropriate.

Once it works with those, if you wanted to support the full feature set, then you could always go back and see about mapping expiration dates less than or equal to WireCache::expireNever to some other searchable property (and using a far future expiration date). But very few people actually use the page/template/selector clearing features (as far as I know), and the core doesn't need them, so they could be considered optional.

Thank you for the detailed explanation! Knowing that some features aren't really used (or at least not in depth) in the core make it appear less daunting. Handling expireNever/expireReserved for saving is pretty straight forward, since it's what Redis does anyway if no expiry is given.

I'll still need to mull over searching and selectors for a bit since a straight forward implementation wouldn't even carry a searchable property - it's really just "cache name" -> "cache value". Storing selectors / template ids will need some kind of helper entries (I'm actually doing that in CacheRedis already but I'm not really happy about it). Or it might even make sense to store expiration selectors / templates and their associated cache names in the MySQL database and just retrieve the cached values themselves from Redis (just thinking aloud here). In any case, I'll give it a try and see if I encounter any stumbling blocks.

[kongondo]

On 5/26/2023 at 8:05 PM, ryan said:

It's not yet clear if there are major benefits one way or the other (cache in database vs. file system), as I've not been able to put all this new code through performance testing yet. I'd definitely be interested to hear if anyone has a chance to test things out. I expect the file system might be faster for reading caches, while the database may be faster for writing caches.

Would it be technically possible and would there be major benefits to decouple 'read' caches versus 'write' caches? I.e., when reading, we use the file system but when writing we use the database system. Perhaps configurable via $config.

Just wondering.

[BitPoet]

On 5/27/2023 at 4:22 PM, ryan said:

This is for the maintenance processes. WireCache performs maintenance every 10 mins and whenever any page or template is saved. The page/template save maintenance only happens if rows have expiry dates prior to WireCache::expireNever. Finding by expiry dates can be optional if the cache (Redis) handles expiration on its own.

I've started implementing my find() and save() methods, and most code is just working around specific selectors by WireCache's built-in maintenance routine to keep my caches consistent. Would it be possible to add a maintenance() method to WireCacheInterface? This could simply return "false" to let WireCache continue using its maintenance logic, or true if maintenance has been dealt with by the cacher instance.

[BitPoet]

2 hours ago, kongondo said:

Would it be technically possible and would there be major benefits to decouple 'read' caches versus 'write' caches? I.e., when reading, we use the file system but when writing we use the database system. Perhaps configurable via $config.

How would the two caches ever match up? You'd have to write your data to both caches at once, which means you'd end up with slow writes anyway.

If you want to use two different caches at the same time for different kinds of data / different areas of the site where one or the other provides real benefits, you can instantiate another WireCache with PW's built-in database engine manually. Assuming you have WireCacheFilesystem installed, you could still do:

<?php namespace ProcessWire;

/* You can put that block in site/ready.php to make $dbCache available everywhere: */
$dbCache = new WireCache();
$dbCache->setCacheModule(new WireCacheDatabase());
wire('dbCache', $dbCache);
/* End of site/ready.php */

// Template code: store a value in the database cache that often gets written but seldom read:
$dbCache->save('testentry1', 'This is saved to the database', 3600);
...
echo $dbCache->get('testentry1');

// Template code: store a value in the filesystem using PW's default engine that gets read
// a lot more often than it is written:
$cache->save('testentry2', 'This is saved in the filesystem', 3600);
...
echo $cache->get('testentry2');

 

[ryan]

@MarkE

Quote

Are you saying that (for example) the selector string option for $cache->save (as per https://processwire.com/api/ref/wire-cache/save/) will no longer be supported? That would be a big problem for me as one of my web apps uses it extensively. 

What I'm intending to communicate is that it's up to the module developer how many of the WireCache features they want to support. That's because, as I understand it, it's not possible to support the the full WireCache feature set with something like Redis. The core WireCache (WireCacheDatabase) will continue to support the full feature set, as it always has. 

For 3rd party WireCache modules, the required feature set would just be the ability to get, save and delete cache by name ... what the majority of cache use cases are. If they can support more than that, great. But I think it's better to keep the door option to more cache options by not requiring specific and lesser-used cache features unique to PW. If a 3rd party module has to support the full WireCache feature set then likely there won't be 3rd party WireCache modules.

For cases like yours, where you are depending on specific features, you'd probably want to stick with using the core default (WireCacheDatabase), unless future 3rd WireCache module is able to support them. 

@BitPoet

Quote

Thank you for the detailed explanation! Knowing that some features aren't really used (or at least not in depth) in the core make it appear less daunting. Handling expireNever/expireReserved for saving is pretty straight forward, since it's what Redis does anyway if no expiry is given.

Great! Yes, that's correct, the core does not use anything but get/save/delete by name. But it does use wildcard matching (for FileCompiler). Does Redis allow for matching by partial cache name? For instance, a cache name that has a wildcard at the end like "MyCache*"? If not, there are ways around it, but I was just curious. 

Quote

Or it might even make sense to store expiration selectors / templates and their associated cache names in the MySQL database and just retrieve the cached values themselves from Redis (just thinking aloud here).

Maybe that is a good way around it. Though having to hit the database in addition to Redis might reduce the benefit, or make it slower than just using the database cache. Maybe another way around it is for WireCache to fallback to the WireCacheDatabase when a particular feature is needed that's not supported by the WireCache module. 

Quote

Would it be possible to add a maintenance() method to WireCacheInterface? This could simply return "false" to let WireCache continue using its maintenance logic, or true if maintenance has been dealt with by the cacher instance.

Good idea, I will add that. 

@kongondo

Quote

Would it be technically possible and would there be major benefits to decouple 'read' caches versus 'write' caches? I.e., when reading, we use the file system but when writing we use the database system. Perhaps configurable via $config.

I don't think so because the cache would have to write and read from the same place, otherwise I don't know how it would find caches that it wrote. But PW does support separate read and write connections for the database (WireCacheDatabase), and it will use them when they are provided. 

 

[BitPoet]

6 minutes ago, ryan said:

But it does use wildcard matching (for FileCompiler). Does Redis allow for matching by partial cache name? For instance, a cache name that has a wildcard at the end like "MyCache*"? If not, there are ways around it, but I was just curious.

It does, but it's usually discouraged, since it raises complexity from O(1) to O(n). I guess as long as the asterisk is added at the end, it might be tolerable in that one instance. I've added a special check for FileCompiler there.

11 minutes ago, ryan said:

Though having to hit the database in addition to Redis might reduce the benefit, or make it slower than just using the database cache. Maybe another way around it is for WireCache to fallback to the WireCacheDatabase when a particular feature is needed that's not supported by the WireCache module. 

I guess it depends on the exact application. If I think of high-load systems with large pages consisting of complex calculations, the benefit in the frontend would mostly still be there, with the biggest workload being (usually) performed in the backend. I'll add both MySQL and Redis implementations anyway to see how they compare.

8 minutes ago, ryan said:

Good idea, I will add that.

Great, that's going to make things infinitely easier. Thank you!

[cb2004]

I am getting a lot of modules saying they still need upgrading after this update, anybody else?

[thetuningspoon]

I'm getting the following error after upgrading:

Fatal Error: Uncaught Error: Class 'FieldtypeFile' not found in wire/core/Modules.php:700

#0 wire/core/Modules.php (1478): Modules->newModule('\Fie...')
#1 wire/core/Fieldtypes.php (199): Modules->getModule('FieldtypeFile')
#2 wire/core/Fields.php (203): Fieldtypes->get('FieldtypeFile')
#3 wire/core/WireSaveableItems.php (260): Fields->makeItem(Array)
#4 wire/core/Fields.php (255): WireSaveableItems->initItem(Array, Object(FieldsArray))
#5 wire/core/WireSaveableItems.php (953): Fields->initItem(Array)
#6 wire/core/WireSaveableItems.php (469): WireSaveableItems->getLazy(434)
#7 wire/core/Fieldgroup.php (127): WireSaveableItems->get('434')
#8 wire/core/Fieldgroup.php (380): Fieldgroup->add('434')
#9 wire/core/WireSaveableItemsLookup.php (135): Fieldgroup->addLookupItem('434', Array)
#10 wire/core/WireSaveableItems.php (953): WireSaveableItemsLookup->initItem(Array)
#11 wire/core/WireSaveableItems.php (469): WireSaveableItems->getLazy(3)
#12 wire/core/Template.php (796): WireSaveableItems->get(3)
#13 wire/core/Templates.php (132): Template->setRaw('fieldgroups_id', 3)
#14 wire/core/WireSaveableItems.php (260): Templates->makeItem(Array)
#15 wire/core/WireSaveableItems.php (953): WireSaveableItems->initItem(Array)
#16 wire/core/WireSaveableItems.php (469): WireSaveableItems->getLazy(3)
#17 wire/core/Templates.php (252): WireSaveableItems->get(3)
#18 wire/core/PagesLoader.php (1241): Templates->get(3)
#19 wire/core/Pages.php (217): PagesLoader->getById(Array)
#20 wire/core/ProcessWire.php (625): Pages->init()
#21 wire/core/ProcessWire.php (582): ProcessWire->initVar('pages', Object(Pages))
#22 wire/core/ProcessWire.php (315): ProcessWire->load(Object(Config))
#23 index.php (52): ProcessWire->__construct(Object(Config))
#24 {main}
thrown (line 700 of wire/core/Modules.php)

I tried emptying the cache folder. FieldtypeFile definitely exists in the wire folder.

I also tried programmatically reloading the modules in ProcessWire.php right before Pages->init() where it fails by calling $wire->modules->reload(). This was the error I got:

Error: Exception: Unable to create path: /reload/ (in wire/core/CacheFile.php line 64)

#0 wire/modules/Markup/MarkupCache.module (103): CacheFile->__construct('/', 'reload', 3600)
#1 wire/core/WireArray.php (2263): MarkupCache->get('reload')
#2 wire/core/WireArray.php (2445): WireArray->explode('reload')
#3 wire/core/Wire.php (419): WireArray->___callUnknown('reload', Array)
#4 wire/core/WireHooks.php (952): Wire->_callMethod('___callUnknown', Array)
#5 wire/core/Wire.php (484): WireHooks->runHooks(Object(Modules), 'callUnknown', Array)
#6 wire/core/Wire.php (487): Wire->__call('callUnknown', Array)
#7 wire/core/ProcessWire.php (582): Wire->__call('reload', Array)
#8 wire/core/ProcessWire.php (315): ProcessWire->load(Object(Config))
#9 index.php (52): ProcessWire->__construct(Object(Config))
#10 {main}

Very strange.

[ryan]

@thetuningspoon  @cb2004 Try the most recent dev branch commits. The "class not found" error at least was coming from upgrading an older version of PW that had certain modules in different parent directories. For instance, /wire/Fieldtype/FieldtypeFile.module was moved to /wire/Fieldtype/FieldtypeFile/FieldtypeFile.module many versions back. But if upgrading from that older version (that had the file in a different location) to 3.0.218 then you'd see this error. At least, that's the issue I ran into (and fixed) here.