Jump to content
MoritzLost

Cache Control - Clear all your caches in one place

Recommended Posts

Process Cache Control

This module provides a simple solution to clearing all your cache layers at once, and an extensible interface to perform various cache-related actions.

The simple motivation behind this module was that I was tired of manually clearing caches in several places after deploying a change on a live site. The basic purpose of this module is a simple Clear all caches link in the Setup menu which clears out all caches, no matter where they hide. You can customize what exactly the module does through it's configuration menu:

  • Expire or delete all cache entries in the database, or selectively clear caches by namespace ($cache API)
  • Clear the the template render cache.
  • Clear out specific folders inside your site's cache directory (/site/assets/cache)
  • Clear the ProCache page render cache (if your site is using ProCache)
  • Refresh version strings for static assets to bust client-side browser caches (this requires some setup, see the full documentation for details).

This is the basic function of the module. However, you can also add different cache management action through the API and execute them through the module's interface. For this advanced usage, the module provides:

  • An interface to see all available cache actions and execute them.
  • A system log and logging output on the module page to see verify what the module is doing.
  • A CacheControlTools class with utility functions to clear out different caches.
  • An API to add cache actions, execute them programmatically and even modify the default action.
  • Permission management, allowing you granular control over which user roles can execute which actions.

The complete documentation can be found in the module's README.

Plans for improvements

If there is some interest in this, I plan to expand this to a more general cache management solution. I particular, I would like to add additional cache actions. Some ideas that came to mind:

  • Warming up the template render cache for publicly accessible pages.
  • Removing all active user sessions.

Let me know if you have more suggestions!

Links

Screenshots

1017168883_Screenshot2020-04-01at13_21_08.thumb.png.075305fb93e3b5aa1f4b41938c36cda0.png

Screenshot 2020-02-06 at 15.23.37.png

  • Like 15

Share this post


Link to post
Share on other sites

Hi @adrian, I didn't know that module existed, it's not in the directory, is it? Because I looked if something like this already existed and didn't find anything. Might've saved me some trouble 😅

That said, I had a quick look, here's what I found in comparing both modules:

  • ClearCacheAdmin exposes more options through the setup page, so there are available to everyone, whereas ProcessCacheControl has it's options in the module configuration, so they are only available to the developer.
  • ClearCacheAdmin has multiple links (and multiple options) on the process page, giving the user more fine control, whereas ProcessCacheControl goes for a simpler interface and bundles all those actions into one (configurable) action with a single button.
  • ProcessCacheControl can delete $cache entries without an expiry date, which ClearCacheAdmin doesn't as far as I can tell. ProcessCacheControl also lets you configure caches to be deleted by namespace, whereas ClearCacheAdmin offers each cache entry to be deleted individually (I think this has mostly to do with the module being a bit older, I believe $cache didn't have all those options back then).
  • The largest difference is my concept of "cache actions", which ClearCacheAdmin doesn't have. I'm not sure how useful that will actually be yet. I think if I can expand on the actions available by default, it will be pretty handy.
    • With ProcessCacheControl, you can add custom cache actions / profiles through the API, that may be useful depending on the use case.
    • Adding to that, ProcessCacheControl has permission checks on a per-action basis.
  • ProcessCacheControl can be modified and executed through the API. In particular, you can modify the default action and execute any action programmatically.
  • Like 5
  • Thanks 1

Share this post


Link to post
Share on other sites

No, it's not in the directory. You should really take a look at @netcarver's awesome: https://pwgeeks.com/ - just type "cache" in there and you'll find some other cache related tools.

Thanks for the rundown of differences - that will be really helpful.

  • Like 5

Share this post


Link to post
Share on other sites

As I was looking as well for a cache clearing solution through the admin and did not find one I just found out about PWGeeks from PW news letter. It seems very nice to have a global list of existing modules, however not everyone would still be aware from the community about the tool, so we might see similar modules being developed based on the customer/developers needs.

I will give a try to both cache clearing modules (if I may generalize the two) and see which one is suiting me more. At least some feedback can be shared.

Thanks @MoritzLost for the new module. It is the play time now 😉

  • Like 1

Share this post


Link to post
Share on other sites

Awesome! ClearCacheAdmin needed some love, this is definitely better IMO.

On 2/7/2020 at 3:12 AM, MoritzLost said:

In particular, can someone let me know if this module causes any problems with the ProCache module?

Will PM you on this, not sure how much I am allowed to talk about it here.

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

Thanks everyone for the feedback! I just pushed a new version that resolves a bug that prevented installs on PHP versions <7.4.
PHP 7.1, 7.2 and 7.3 should work now!

Thanks @Mikie for your help! I have also found some documentation in the ProCache store page. I've added an option to clear the ProCache now as well, though I haven't included it in the new version because I can't test it out. Would you mind giving it a go? The ProCache integration is in the procache branch on Github (download link). If it's working as intended, it should add a new option to clear the ProCache in the module configuration, including a note saying whether ProCache is installed. If selected, the 'Clear all' action should use $procache->clearAll and log the result. It would also be good to know if non-superusers can use this, or if it's limited to the superuser. Thanks!

The module also appears to have been approved to the module directory, so it can now be downloaded directly through the backend using the module name ProcessCacheControl. Thanks @adrian (or is Ryan in charge of the module directory? 🙃)

  • Like 6

Share this post


Link to post
Share on other sites

Good thing I marked this as a beta release, because I'm still finding a lot of bugs 😅

I just pushed version 0.4.0. The module is now no longer autoloaded, because it's not really required anymore. This release splits the module into two seperate Modules: ProcessCacheControl and CacheControlTools. They are still installed alongside each other, and ProcessCacheControl will install CacheControlTools during install. For existing installs, you might have to reinstall the module, or install CacheControlTools manually after the upgrade. Also, the asset version management is now part of CacheControlTools, so the code has to be adjusted:

// version 0.3.0 and below
$CacheControlTools = $modules->get('ProcessCacheControl')->getTools();

// version 0.4.0
$CacheControlTools = $modules->get('CacheControlTools');

A bit of background for the split: I learned something new about the permission system. If a Process module defines a permission, the current user needs that permission not only to access the Process page in the admin, but also for the module to even be instantiated at all. Because of this, the asset management system didn't work for guest users. By splitting the repository into two modules, I can still have strict access control for the Process part of the module, while allowing the tools module to be instantiated even during requests from guest users.

Anyway, the module is now, let's say, nearing stability. I'd like to publish a stable release soon, so let me know if you find any more bugs!

  • Like 4

Share this post


Link to post
Share on other sites

I just updated the module to version 0.5.0! I have now installed it on a couple of sites and it's pretty stable by now. But before I tag a stable 1.0.0 release I would really like to get the ProCache integration working.

ProCache integration - Testers wanted

If somebody has access to the ProCache module and would like to help me, I just need someone to test the ProCache integration!
To get started, download the procache branch of the module on Github and install it . There should be a new "ProCache" option in the module configuration. This field should also tell you whether you have ProCache installed. Check this option. Then go to the Process page through Setup -> Cache Control and click the "Clear all" button. The log output should clear the ProCache entirely. I'd like to know if you get any errors during any of those steps. Also, please check if the ProCache has been actually cleared.

Thanks!

Version 0.5.0

The new version provides some additional helper methods to check if users can access particular actions and improves the status messages displayed on the process page. In particular, there's a new static method to check if the user can access the module. This is important, because instantiating the module ($modules->get('ProcessCacheControl')) will throw an error if the user does not have the required permission. So this can now be checked with the static method ProcessCacheControl::canUseModule($user).

The complete Changelog can be found in the CHANGELOG.md in the repository.

  • Like 2

Share this post


Link to post
Share on other sites
5 hours ago, MoritzLost said:

ProCache integration - Testers wanted

I successfully tested it on one site several times (ca. 10x) with ProCaches between 50 and 700 pages! 🙂

Nice work!

  • Thanks 1

Share this post


Link to post
Share on other sites
12 hours ago, horst said:

I successfully tested it on one site several times (ca. 10x) with ProCaches between 50 and 700 pages! 🙂

Nice work!

@horst Awesome, thank you very much! I'll merge the branch into master soon then. I just need to update the documentation with all the new methods I added in the previous release, then I'll update the module to the first stable release.

I'm still thinking about other useful cache actions. I considered a "warm up" method that will just iterate over all the pages and perform anonymous HTTP requests to warm up the template render cache. Does something like this already exist? Would it be useful to anyone?
For smaller sites it would be trivial, but for larger sites the module might need some form of rate limiting, batch processing ... not sure if it's worth the effort?

  • Like 1

Share this post


Link to post
Share on other sites
59 minutes ago, MoritzLost said:

I considered a "warm up" method that will just iterate over all the pages and perform anonymous HTTP requests to warm up the template render cache. Does something like this already exist? Would it be useful to anyone?
For smaller sites it would be trivial, but for larger sites the module might need some form of rate limiting, batch processing ... not sure if it's worth the effort?

@MoritzLost A warmUp cache is definetly on the wishlist of some of us. This came up from time to time in forum posts. But with ProCache and larger sites it definetly is not trivial.

For smaller sites I have build some individual (hacky) solutions to drop and rebuild the ProCache pages, but for larger sites I manually use a local program that traverses a given website and collect information about broken links, redirects, etc.

I'm not sure how big the effort will / must be. I do know some sites (where I use my local tool) that have large archive content, that never will change. My PC time cache limit there is one year! It also could be infinite.

  • Like 2

Share this post


Link to post
Share on other sites

@horst Thanks for the feedback. Does warming up ProCache require any special consideration? I liked the approach of HTTP requests because of it's simplicity – i.e. it's virtually identical with a regular page visits, so the normal caching mechanisms can just do their thing. Or will that not work with ProCache for some reason?

Share this post


Link to post
Share on other sites

@MoritzLost I think it is "the best way!" to use (guest) HTTP requests to warm up a ProCache.  Thats why I really like the use of this crawler tool. It starts at the homepage, determine links to the homepages childpages, fetch and parse them, and so on.

I think the real problem is to get all "real public URLs" from within PW, as I have sites that use many many pages that have a public state, but in real they are only single sections of a page. So, maybe thats my private problem, but I'm not sure if there is a way to detect which pages / parent pages are official / public URLs. I think if a warmup function can get passed a optional URL list or a PageArray, that overwrites a default behavior, this would be a great benefit for all out there, that uses MVC and other individual approaches. 

  • Like 1

Share this post


Link to post
Share on other sites

@horst Hm, I hadn't considered that, that makes it more difficult indeed. Come to think of it, most of my sites would have this issue as well - for example, I use a non-standard pagination approach with URL segments, those wouldn't be caught by just using the public URL of the base page; only the first page would be reached. I guess webcrawling would be a way around that, though that is a real can of worms ...

Maybe a good compromise would be making the process hookable, like you said. Maybe the method can call a hookable method that returns an array of URLs for each page; by default it returns only the normal URLs, and additional URLs with additional parameters, url segments et c. can be added through hooks. I'll give this a go when I have the time!

  • Like 1

Share this post


Link to post
Share on other sites
1 hour ago, MoritzLost said:

though that is a real can of worms ...

😆

Share this post


Link to post
Share on other sites
8 hours ago, MoritzLost said:

I considered a "warm up" method that will just iterate over all the pages and perform anonymous HTTP requests to warm up the template render cache. Does something like this already exist? Would it be useful to anyone?

I think all caching "solutions" would somehow benefit with "warm-up" (you can try a new version of AIOM that mimics how ProCache works). Most crawlers do what you want to do more or less successfully (with some gotchas you and Horst mentioned), but it would certainly look more "professional" and it would be integrated into PW.

  • Like 2

Share this post


Link to post
Share on other sites
On 3/19/2020 at 4:09 PM, MoritzLost said:

ProCache integration - Testers wanted

@MoritzLost Nice work! Tested: the page cache is deleted, but not the compiled assets (js, css) of ProCache. PW DEV 3.0.153, ProCache latest, Win and Linux, PHP 7.3, PHP 7.4

  • Like 1

Share this post


Link to post
Share on other sites
2 hours ago, David Karich said:

@MoritzLost Nice work! Tested: the page cache is deleted, but not the compiled assets (js, css) of ProCache. PW DEV 3.0.153, ProCache latest, Win and Linux, PHP 7.3, PHP 7.4

Thanks @David Karich! Hm, right now the module is just calling $procache->clearAll() which is supposed to clear everything according to the ProCache store page. Is there another method to clear the minified assets as well? I checked all documentation I could find, a couple of ideas:

  1. I don't know how ProCache works with minified assets, but according to this blog post any changes to the CSS & JS files should be picked up automatically, so maybe it's not necessary to clear those? Though of course that is not ideal for a cache clear button 🙂
  2. Judging by the last screenshot on this blog post, ProCache stores minified assets inside /site/assets/pwpc, or is that out of date? Currently, ProcessCacheControl only supports clearing out folders inside /site/assets/cache (so it can't accidentally nuke the files directory because of a misconfigured setting ...). Maybe I could extend that to also allow clearing /site/assets/pwpc.  Is that directory path static or can it be changed?

Anyway, I'll put out a stable release with updated documentation & the ProCache integration soon! Hopefully the minification issue can also be resolved then.

  • Like 1

Share this post


Link to post
Share on other sites

I have now integrated the ProCache branch into the module and released it as version 1.0.0!

  • I have tested the module on multiple sites and it's pretty stable now. But if you do run into bugs, let me know!
  • The ProCache integration is now included, so you can clear out the ProCache page render cache alongside all other caches. Hopefully we can resolve the issue with minified assets, see the previous post.
  • The documentation is now updated with information on all the new methods added in the previous release.

Let me know if something isn't working for you!

  • Like 4

Share this post


Link to post
Share on other sites

@MoritzLost - huge thanks! I needed exactly such a link/module to clear all kind of cache. 

PW 3.0.156, PHP 7.4, OpenLiteSpeed - works like a charm. 👍

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By MoritzLost
      This module allows you to integrate hCaptcha bot / spam protection into ProcessWire forms. hCaptcha is a great alternative to Google ReCaptcha, especially if you are in the EU and need to comply with privacy regulations.

      The development of this module is sponsored by schwarzdesign.
      The module is built as an Inputfield, allowing you to integrate it into any ProcessWire form you want. It's primarily intended for frontend forms and can be added to Form Builder forms for automatic spam protection. There's a step-by-step guide for adding the hCaptcha widget to Form Builder forms in the README, as well as instructions for API usage.
      Features
      Inputfield that displays an hCaptcha widget in ProcessWire forms. The inputfield verifies the hCaptcha response upon submission, and adds a field error if it is invalid. All hCaptcha configuration options for the widget (theme, display size etc) can be changed through the inputfield configuration, as well as programmatically. hCaptcha script options can be changed through a hook. Error messages can be translated through ProcessWire's site translations. hCaptcha secret keys and site-keys can be set for each individual inputfield or globally in your config.php. Error codes and failures are logged to help you find configuration errors. Please check the README for setup instructions.
      Links
      Github Repository and documentation InputfieldHCaptcha in the module directory (pending approval) Screenshots (configuration)

      Screenshots (hCaptcha widget)

       
       

       
    • By joshua
      This module is (yet another) way for implementing a cookie management solution.
      Of course there are several other possibilities:
      - https://processwire.com/talk/topic/22920-klaro-cookie-consent-manager/
      - https://github.com/webmanufaktur/CookieManagementBanner
      - https://github.com/johannesdachsel/cookiemonster
      - https://www.oiljs.org/
      - ... and so on ...
      In this module you can configure which kind of cookie categories you want to manage:

      You can also enable the support for respecting the Do-Not-Track (DNT) header to don't annoy users, who already decided for all their browsing experience.
      Currently there are four possible cookie groups:
      - Necessary (always enabled)
      - Statistics
      - Marketing
      - External Media
      All groups can be renamed, so feel free to use other cookie group names. I just haven't found a way to implement a "repeater like" field as configurable module field ...
      When you want to load specific scripts ( like Google Analytics, Google Maps, ...) only after the user's content to this specific category of cookies, just use the following script syntax:
      <script type="text/plain" data-type="text/javascript" data-category="statistics" data-src="/path/to/your/statistic/script.js"></script> <script type="text/plain" data-type="text/javascript" data-category="marketing" data-src="/path/to/your/mareketing/script.js"></script> <script type="text/plain" data-type="text/javascript" data-category="external_media" data-src="/path/to/your/external-media/script.js"></script> <script type="text/plain" data-type="text/javascript" data-category="marketing">console.log("Inline scripts are also working!");</script> The type has to be "optin" to get recognized by PrivacyWire, the data-attributes are giving hints, how the script shall be loaded, if the data-category is within the cookie consents of the user. These scripts are loaded asynchronously after the user made the decision.
      If you want to give the users the possibility to change their consent, you can use the following Textformatter:
      [[privacywire-choose-cookies]] It's planned to add also other Textformatters to opt-out of specific cookie groups or delete the whole consent cookie.
      You can also add a custom link to output the banner again with a link / button with following class:
      <a href="#" class="privacywire-show-options">Show Cookie Options</a> <button class="privacywire-show-options">Show Cookie Options</button> This module is still in development, but we already use it on several production websites.
      You find it here: PrivacyWire Git Repo
      Download as .zip
      I would love to hear your feedback 🙂
      CHANGELOG
      0.1.1 Debugging: fixed error during uninstall 0.1.0 Added new detection of async scripts for W3C Validation 0.0.6 CSS-Debugging for hiding unused buttons, added ProCache support for the JavaScript tag 0.0.5 Multi-language support included completely (also in TextFormatter). Added possibility to async load other assets (e.g. <img type="optin" data-category="marketing" data-src="https://via.placeholder.com/300x300">) 0.0.4 Added possibility to add an imprint link to the banner 0.0.3 Multi-language support for module config (still in development) 0.0.2 First release 0.0.1 Early development
    • By bernhard
      --- Please use RockFinder3 ---
    • By MoritzLost
      Cacheable Placeholders
      This module allows you to have pieces of dynamic content inside cached output. This aims to solve the common problem of having a mostly cacheable site, but with pieces of dynamic output here and there.  Consider this simple example, where you want to output a custom greeting to the current user:
      <h1>Good morning, <?= ucfirst($user->name) ?></h1> This snippet means you can't use the template cache (at least for logged-in users), because each user has a different name. Even if 99% of your output is static, you can only cache the pieces that you know won't include this personal greeting. A more common example would be CSRF tokens for HTML forms - those need to be unique by definition, so you can't cache the form wholesale.
      This module solves this problem by introducing cacheable placeholders - small placeholder tokens that get replaced during every request. The replacement is done inside a Page::render hook so it runs during every request, even if the response is served from the template cache. So you can use something like this:
      <h1>Good morning, {{{greeting}}}</h1> Replacement tokens are defined with a callback function that produces the appropriate output and added to the module through a simple hook:
      // site/ready.php wire()->addHookAfter('CachePlaceholders::getTokens', function (HookEvent $e) { $tokens = $e->return; $tokens['greeting'] = [ 'callback' => function (array $tokenData) { return ucfirst(wire('user')->name); } ]; $e->return = $tokens; }); Tokens can also include parameters that are parsed and passed to the callback function. There are more fully annotated examples and step-by-step instructions in the README on Github!
      Features
      A simple and fast token parser that calls the appropriate callback and runs automatically. Tokens may include multiple named or positional parameters, as well as multi-value parameters. A manual mode that allows you to replace tokens in custom pieces of cached content (useful if you're using the $cache API). Some built-in tokens for common use-cases: CSRF-Tokens, replacing values from superglobals and producing random hexadecimal strings. The token format is completely customizable, all delimiters can be changed to avoid collisions with existing tag parsers or template languages. Links
      Github Repository & documentation Module directory (pending approval) If you are interested in learning more, the README is very extensive, with more usage examples, code samples and usage instructions!
×
×
  • Create New...