module Cache Control - Clear all your caches in one place

[MoritzLost]

Process Cache Control

This module provides a simple solution to clearing all your cache layers at once, and an extensible interface to perform various cache-related actions.

The simple motivation behind this module was that I was tired of manually clearing caches in several places after deploying a change on a live site. The basic purpose of this module is a simple Clear all caches link in the Setup menu which clears out all caches, no matter where they hide. You can customize what exactly the module does through it's configuration menu:

• Expire or delete all cache entries in the database, or selectively clear caches by namespace ($cache API)
• Clear the the template render cache.
• Clear out specific folders inside your site's cache directory (/site/assets/cache)
• Clear the ProCache page render cache (if your site is using ProCache)
• Refresh version strings for static assets to bust client-side browser caches (this requires some setup, see the full documentation for details).

This is the basic function of the module. However, you can also add different cache management action through the API and execute them through the module's interface. For this advanced usage, the module provides:

• An interface to see all available cache actions and execute them.
• A system log and logging output on the module page to see verify what the module is doing.
• A CacheControlTools class with utility functions to clear out different caches.
• An API to add cache actions, execute them programmatically and even modify the default action.
• Permission management, allowing you granular control over which user roles can execute which actions.

The complete documentation can be found in the module's README.

Plans for improvements

If there is some interest in this, I plan to expand this to a more general cache management solution. I particular, I would like to add additional cache actions. Some ideas that came to mind:

• Warming up the template render cache for publicly accessible pages.
• Removing all active user sessions.

Let me know if you have more suggestions!

Links

• https://github.com/MoritzLost/ProcessCacheControl
• ProcessCacheControl in the Module directory
• CHANGELOG in the repository

Screenshots

[adrian]

Hi @MoritzLost - this looks great. I wonder if you could provide a quick summary of the differences with https://github.com/somatonic/ClearCacheAdmin

Thanks again!

[MoritzLost]

Hi @adrian, I didn't know that module existed, it's not in the directory, is it? Because I looked if something like this already existed and didn't find anything. Might've saved me some trouble ?

That said, I had a quick look, here's what I found in comparing both modules:

• ClearCacheAdmin exposes more options through the setup page, so there are available to everyone, whereas ProcessCacheControl has it's options in the module configuration, so they are only available to the developer.
• ClearCacheAdmin has multiple links (and multiple options) on the process page, giving the user more fine control, whereas ProcessCacheControl goes for a simpler interface and bundles all those actions into one (configurable) action with a single button.
• ProcessCacheControl can delete $cache entries without an expiry date, which ClearCacheAdmin doesn't as far as I can tell. ProcessCacheControl also lets you configure caches to be deleted by namespace, whereas ClearCacheAdmin offers each cache entry to be deleted individually (I think this has mostly to do with the module being a bit older, I believe $cache didn't have all those options back then).
• The largest difference is my concept of "cache actions", which ClearCacheAdmin doesn't have. I'm not sure how useful that will actually be yet. I think if I can expand on the actions available by default, it will be pretty handy.
  • With ProcessCacheControl, you can add custom cache actions / profiles through the API, that may be useful depending on the use case.
  • Adding to that, ProcessCacheControl has permission checks on a per-action basis.
• ProcessCacheControl can be modified and executed through the API. In particular, you can modify the default action and execute any action programmatically.

[adrian]

No, it's not in the directory. You should really take a look at @netcarver's awesome: https://pwgeeks.com/ - just type "cache" in there and you'll find some other cache related tools.

Thanks for the rundown of differences - that will be really helpful.

[dragan]

8 minutes ago, adrian said:

You should really take a look at @netcarver's awesome: https://pwgeeks.com/ 

It's a shame though he doesn't allow search engines to index the site: https://pwgeeks.com/robots.txt

</derail>

[MilenKo]

As I was looking as well for a cache clearing solution through the admin and did not find one I just found out about PWGeeks from PW news letter. It seems very nice to have a global list of existing modules, however not everyone would still be aware from the community about the tool, so we might see similar modules being developed based on the customer/developers needs.

I will give a try to both cache clearing modules (if I may generalize the two) and see which one is suiting me more. At least some feedback can be shared.

Thanks @MoritzLost for the new module. It is the play time now ?

[Mikie]

Awesome! ClearCacheAdmin needed some love, this is definitely better IMO.

On 2/7/2020 at 3:12 AM, MoritzLost said:

In particular, can someone let me know if this module causes any problems with the ProCache module?

Will PM you on this, not sure how much I am allowed to talk about it here.

[MoritzLost]

Thanks everyone for the feedback! I just pushed a new version that resolves a bug that prevented installs on PHP versions <7.4.
PHP 7.1, 7.2 and 7.3 should work now!

Thanks @Mikie for your help! I have also found some documentation in the ProCache store page. I've added an option to clear the ProCache now as well, though I haven't included it in the new version because I can't test it out. Would you mind giving it a go? The ProCache integration is in the procache branch on Github (download link). If it's working as intended, it should add a new option to clear the ProCache in the module configuration, including a note saying whether ProCache is installed. If selected, the 'Clear all' action should use $procache->clearAll and log the result. It would also be good to know if non-superusers can use this, or if it's limited to the superuser. Thanks!

The module also appears to have been approved to the module directory, so it can now be downloaded directly through the backend using the module name ProcessCacheControl. Thanks @adrian (or is Ryan in charge of the module directory? ?)

[MoritzLost]

Good thing I marked this as a beta release, because I'm still finding a lot of bugs ?

I just pushed version 0.4.0. The module is now no longer autoloaded, because it's not really required anymore. This release splits the module into two seperate Modules: ProcessCacheControl and CacheControlTools. They are still installed alongside each other, and ProcessCacheControl will install CacheControlTools during install. For existing installs, you might have to reinstall the module, or install CacheControlTools manually after the upgrade. Also, the asset version management is now part of CacheControlTools, so the code has to be adjusted:

// version 0.3.0 and below
$CacheControlTools = $modules->get('ProcessCacheControl')->getTools();

// version 0.4.0
$CacheControlTools = $modules->get('CacheControlTools');

A bit of background for the split: I learned something new about the permission system. If a Process module defines a permission, the current user needs that permission not only to access the Process page in the admin, but also for the module to even be instantiated at all. Because of this, the asset management system didn't work for guest users. By splitting the repository into two modules, I can still have strict access control for the Process part of the module, while allowing the tools module to be instantiated even during requests from guest users.

Anyway, the module is now, let's say, nearing stability. I'd like to publish a stable release soon, so let me know if you find any more bugs!

[MoritzLost]

I just updated the module to version 0.5.0! I have now installed it on a couple of sites and it's pretty stable by now. But before I tag a stable 1.0.0 release I would really like to get the ProCache integration working.

ProCache integration - Testers wanted

If somebody has access to the ProCache module and would like to help me, I just need someone to test the ProCache integration!
To get started, download the procache branch of the module on Github and install it . There should be a new "ProCache" option in the module configuration. This field should also tell you whether you have ProCache installed. Check this option. Then go to the Process page through Setup -> Cache Control and click the "Clear all" button. The log output should clear the ProCache entirely. I'd like to know if you get any errors during any of those steps. Also, please check if the ProCache has been actually cleared.

Thanks!

Version 0.5.0

The new version provides some additional helper methods to check if users can access particular actions and improves the status messages displayed on the process page. In particular, there's a new static method to check if the user can access the module. This is important, because instantiating the module ($modules->get('ProcessCacheControl')) will throw an error if the user does not have the required permission. So this can now be checked with the static method ProcessCacheControl::canUseModule($user).

The complete Changelog can be found in the CHANGELOG.md in the repository.

[horst]

5 hours ago, MoritzLost said:

ProCache integration - Testers wanted

I successfully tested it on one site several times (ca. 10x) with ProCaches between 50 and 700 pages! ?

Nice work!

[MoritzLost]

12 hours ago, horst said:

I successfully tested it on one site several times (ca. 10x) with ProCaches between 50 and 700 pages! ?

Nice work!

@horst Awesome, thank you very much! I'll merge the branch into master soon then. I just need to update the documentation with all the new methods I added in the previous release, then I'll update the module to the first stable release.

I'm still thinking about other useful cache actions. I considered a "warm up" method that will just iterate over all the pages and perform anonymous HTTP requests to warm up the template render cache. Does something like this already exist? Would it be useful to anyone?
For smaller sites it would be trivial, but for larger sites the module might need some form of rate limiting, batch processing ... not sure if it's worth the effort?

[horst]

59 minutes ago, MoritzLost said:

I considered a "warm up" method that will just iterate over all the pages and perform anonymous HTTP requests to warm up the template render cache. Does something like this already exist? Would it be useful to anyone?
For smaller sites it would be trivial, but for larger sites the module might need some form of rate limiting, batch processing ... not sure if it's worth the effort?

@MoritzLost A warmUp cache is definetly on the wishlist of some of us. This came up from time to time in forum posts. But with ProCache and larger sites it definetly is not trivial.

For smaller sites I have build some individual (hacky) solutions to drop and rebuild the ProCache pages, but for larger sites I manually use a local program that traverses a given website and collect information about broken links, redirects, etc.

I'm not sure how big the effort will / must be. I do know some sites (where I use my local tool) that have large archive content, that never will change. My PC time cache limit there is one year! It also could be infinite.

[MoritzLost]

@horst Thanks for the feedback. Does warming up ProCache require any special consideration? I liked the approach of HTTP requests because of it's simplicity – i.e. it's virtually identical with a regular page visits, so the normal caching mechanisms can just do their thing. Or will that not work with ProCache for some reason?

[horst]

@MoritzLost I think it is "the best way!" to use (guest) HTTP requests to warm up a ProCache.  Thats why I really like the use of this crawler tool. It starts at the homepage, determine links to the homepages childpages, fetch and parse them, and so on.

I think the real problem is to get all "real public URLs" from within PW, as I have sites that use many many pages that have a public state, but in real they are only single sections of a page. So, maybe thats my private problem, but I'm not sure if there is a way to detect which pages / parent pages are official / public URLs. I think if a warmup function can get passed a optional URL list or a PageArray, that overwrites a default behavior, this would be a great benefit for all out there, that uses MVC and other individual approaches. 

[MoritzLost]

@horst Hm, I hadn't considered that, that makes it more difficult indeed. Come to think of it, most of my sites would have this issue as well - for example, I use a non-standard pagination approach with URL segments, those wouldn't be caught by just using the public URL of the base page; only the first page would be reached. I guess webcrawling would be a way around that, though that is a real can of worms ...

Maybe a good compromise would be making the process hookable, like you said. Maybe the method can call a hookable method that returns an array of URLs for each page; by default it returns only the normal URLs, and additional URLs with additional parameters, url segments et c. can be added through hooks. I'll give this a go when I have the time!

[horst]

1 hour ago, MoritzLost said:

though that is a real can of worms ...

?

[matjazp]

8 hours ago, MoritzLost said:

I considered a "warm up" method that will just iterate over all the pages and perform anonymous HTTP requests to warm up the template render cache. Does something like this already exist? Would it be useful to anyone?

I think all caching "solutions" would somehow benefit with "warm-up" (you can try a new version of AIOM that mimics how ProCache works). Most crawlers do what you want to do more or less successfully (with some gotchas you and Horst mentioned), but it would certainly look more "professional" and it would be integrated into PW.

[David Karich]

On 3/19/2020 at 4:09 PM, MoritzLost said:

ProCache integration - Testers wanted

@MoritzLost Nice work! Tested: the page cache is deleted, but not the compiled assets (js, css) of ProCache. PW DEV 3.0.153, ProCache latest, Win and Linux, PHP 7.3, PHP 7.4

[MoritzLost]

2 hours ago, David Karich said:

@MoritzLost Nice work! Tested: the page cache is deleted, but not the compiled assets (js, css) of ProCache. PW DEV 3.0.153, ProCache latest, Win and Linux, PHP 7.3, PHP 7.4

Thanks @David Karich! Hm, right now the module is just calling $procache->clearAll() which is supposed to clear everything according to the ProCache store page. Is there another method to clear the minified assets as well? I checked all documentation I could find, a couple of ideas:

1. I don't know how ProCache works with minified assets, but according to this blog post any changes to the CSS & JS files should be picked up automatically, so maybe it's not necessary to clear those? Though of course that is not ideal for a cache clear button ?
2. Judging by the last screenshot on this blog post, ProCache stores minified assets inside /site/assets/pwpc, or is that out of date? Currently, ProcessCacheControl only supports clearing out folders inside /site/assets/cache (so it can't accidentally nuke the files directory because of a misconfigured setting ...). Maybe I could extend that to also allow clearing /site/assets/pwpc.  Is that directory path static or can it be changed?

Anyway, I'll put out a stable release with updated documentation & the ProCache integration soon! Hopefully the minification issue can also be resolved then.

[MoritzLost]

I have now integrated the ProCache branch into the module and released it as version 1.0.0!

• I have tested the module on multiple sites and it's pretty stable now. But if you do run into bugs, let me know!
• The ProCache integration is now included, so you can clear out the ProCache page render cache alongside all other caches. Hopefully we can resolve the issue with minified assets, see the previous post.
• The documentation is now updated with information on all the new methods added in the previous release.

Let me know if something isn't working for you!

[Chris Ernovsky]

@MoritzLost - huge thanks! I needed exactly such a link/module to clear all kind of cache. 

PW 3.0.156, PHP 7.4, OpenLiteSpeed - works like a charm. ?

[MoritzLost]

ProcessCacheControl version 1.1.0 released

I just released an updated version of this module which fixes some issues with the asset version management feature. In particular:

• When using CacheControlTools::getAssetVersion while no asset version existed (for the specified category), the implicitly generated version incorrectly expired after one day. This is now fixed, so asset versions will not expirate anymore until manually cleared.
• Asset versions are no stored using WireCache::expireNever instead of WireCache::expireReserved. This means they will be deleted when using $cache->deleteAll(), when previously you had to explicity specify them by namespace to clear the cached versions.

Links:

• Full changelog for version 1.1.0
• ProcessCacheControl in the modules directory

[gebeer]

@MoritzLost is your module ready for PHP 8?

[MoritzLost]

6 hours ago, gebeer said:

@MoritzLost is your module ready for PHP 8?

@gebeer We're running the module on a couple of sites on PHP 8.0 and 8.1 with no issues. If you do find any issues running on PHP 8+, open an issue on GitHub or let me know here and I will fix it as soon as possible!