Leaderboard

Continuing from last week's post and discussion, ProcessWire 3.0.218 decouples the modules system from the cache system. Now the modules system maintains its own internal caches (at least once you do a Modules > Refresh). It'll still use the $cache API as a backup (temporarily), but now you can safely export the database without the "caches" table, or even delete the "caches" table, if you want to. It'll get re-created as needed. In this version, work also continued on the new WireCacheInterface (and major updates in WireCache) so that we could support external modules to handle cache storage. This capability is kind of similar to how we support 3rd party WireMail and WireSessionHandler modules. The first example is WireCacheDatabase, which is the default cache storage handler for the core. And today we have a new module called WireCacheFilesystem that replaces the default WireCache database storage with a file-system based storage, once installed. It's not yet clear if there are major benefits one way or the other (cache in database vs. file system), as I've not been able to put all this new code through performance testing yet. I'd definitely be interested to hear if anyone has a chance to test things out. I expect the file system might be faster for reading caches, while the database may be faster for writing caches. At least that's what I found with a few preliminary experiments, but they haven't been very thorough, so take that with a grain of salt. I thought we needed at least 2 examples of classes implementing WireCacheInterface before we'd be ready to support potential 3rd party WireCache modules. I imagine that 3rd party modules getting into dedicated cache options independent of database or file system is where we'll start to see major performance benefits. At least for sites that use the cache heavily. That's all for this week, have a great weekend!

About that: I'm itching (as far as time allows) to turn CacheRedis into a WireCacheInterface compatible module, but there might be a few stumbling blocks. I see that your modules support searching by expiry, which is not something most key-value-stores are designed for since they take care of expiry on their own. It's okay and possible to read the expiration value for an individual key, though, so I'm not completely sure yet how much of an impact a lack of that is going to have, and which applications besides maintenance are affected. More of a headache will be handling expiration constants like WireCache::expireNever, WireCache::expireReserved etc. since they hold dates in the past, meaning entries with those would expire the moment they are written. My gut feeling tells me that a lot more refactoring in WireCache will be necessary to decouple most of the expiration logic from the cacher implementation, but it might be possible to handle most cases by intercepting those values and applying a different logic. Not sure if I'll find time this (long, here in Germany) weekend to take a closer look. Have a great weekend as well!

Ahh! Reading this in the early morning laying in bed. Almost always start the weekend with PW blog. What I read makes me smile. A great weekend indeed! Thanks!

@BitPoet This is for the maintenance processes. WireCache performs maintenance every 10 mins and whenever any page or template is saved. The page/template save maintenance only happens if rows have expiry dates prior to WireCache::expireNever. Finding by expiry dates can be optional if the cache (Redis) handles expiration on its own. I'm also not sure how much need for these two constants anymore since the modules are now decoupled from the cache. Though I think TracyDebugger uses expireNever. You can just replace WireCache::expireNever with some date far in the future (a year?). The WireCache::expireReserved can be treated the same as expireNever. It was primarily for the modules-system and I will likely deprecate this option since it's no longer needed. Its only purpose is to behave like expireNever but also prevent the rows from being deleted when a "delete all" action is being performed. We don't need that anymore. There are a few expirations prior to expireNever that could also considered. Either that, or the WireCacheInterface module could just indicate it doesn't support them. Since these are all dates before WireCache::expireNever, they can be easily filtered in the same way. If the cache can't search by expiry, then implementing them would mean mapping to some other property that is searchable by the cache handler. Any WireCacheInterface module doesn't need to know about any of these since all the logic for finding them is mapped to expiration dates: WireCache::expireSave - This expiration means that the cache row should be deleted whenever any Page or Template is saved. WireCache::expireSelector - This is like expireSave above except that it indicates the cache value is JSON-encoded along with a property named 'selector' and this contains a page-finding selector. WireCache loads them all and when the saved page matches any selector then the cache row is deleted. WireCache maps template IDs to dates by doing $expire=date('Y-m-d H:i:s', $template->id); and that can be reversed with $templateID=strtotime($expire); When a page having that template ID is saved, the cache row having that templateID-based expire date is deleted. Like with the above, the WireCacheInterface module doesn't need to know anything about this other than finding rows having those expiration dates, and presumably those could just be mapped to something else. I'm not sure it's necessary to support these other than mapping expireNever to a future date, as I think very few actually use these features. From what you've described, Redis handles a lot of the stuff that WireCache does on its own. If the cache handler (Redis) can handle these tasks on its own, that's great. If I'm understanding what you've described correctly, here's how I would handle it: When find($options) receives a request containing $options['names'] to find, then ignore anything in the $options['expires'], since Redis is never going to return an expired row. When find($options) receives a request with empty $options['names'], and a populated $options['expires'], just return a blank array (no results). When save() receives an $expire date of expireNever or expireReserved, substitute some far future date instead. When save() receives an $expire date prior to expireReserved, throw an Exception to say the feature isn't supported, or substitute 1 hour, 1 day, or whatever you think is appropriate. Once it works with those, if you wanted to support the full feature set, then you could always go back and see about mapping expiration dates less than or equal to WireCache::expireNever to some other searchable property (and using a far future expiration date). But very few people actually use the page/template/selector clearing features (as far as I know), and the core doesn't need them, so they could be considered optional.

I'm both embarrassed to say that I didn't even realize this was here (despite it being front-and-center in the UI) and elated to discover it today thanks to this sentence. ♥ PW has so many (amazing) things included, it's hard to keep track. Even though you've built it all, I'm surprised you're able to keep track of them all too!!

Following up from the module mentioned last week, the PageEditRestore module has been released and here's a new blog post with all the details. This module helps to prevent page edits in the admin from getting lost when the user’s session is lost— https://processwire.com/blog/posts/page-edit-restore-module/

This week ProcessWire 3.0.217 is released with 10 issue fixes, 2 PRs and a couple of minor additions too. See the dev branch changelog for details. Recently a client called me in a panic because they'd spent a few hours making edits to a page, and when they finally hit save, they were no longer logged in, so their changes were seemingly lost. I guess that their IP had changed somehow, or they kept the page editor open overnight or something. Whatever it was, they were now sitting at the login screen with their changes apparently lost forever. Luckily this person left that window as-is and contacted me to see if there was any way I could recover their changes. I quickly edited their /site/config.php file and temporarily added these: $config->protectCSRF = false; $config->sessionFingerprint = false; Next, I asked them to open another tab and login there. Once logged in, they returned to the tab where the page save failed, then hit "reload" in their browser, and their changes were saved. Phew. Thankfully that worked, but if it didn't, the next thing we were going to try was to open the browser inspector "Network" tab, and then copy/paste the edited content right out of the browser's POST data and into the CKEditor HTML source window. I imagine this has happened to others and perhaps they weren't so lucky as to recover the unsaved changes. So how can you avoid this issue? The best bet is to just save your work regularly. But that doesn't always happen, no matter how many times we communicate that to the client. So you can reduce the probability of it by making a couple adjustments to your config.php file. One change would be increasing your $config->sessionExpireSeconds. But the default is already 86400 seconds (1 day), and I'm not sure many really take more than a day between starting an edit and saving it... though I'm sure it happens. Another change would be turning off the $config->sessionFingerprint (or loosening it, see fingerprint settings). That's trading security for convenience, which isn't ideal, but it would prevent a changed IP address from expiring the session. Another thing you can do is install the ProDevTools UserActivity module, which keeps a ping going to the server, preventing you from getting logged out due to inactivity. Though this doesn't prevent a changed session fingerprint from logging you out, though it at least alerts you as soon as you've been logged out. Even the above changes might not completely solve this issue, and I don't like to tailor session settings around this case either (reducing security), so I've been thinking of alternatives. After dwelling on it for awhile, I started working on a module that saves non-authenticated POST requests sent to the page editor... saving data that would otherwise get lost. Then when you go back to edit the page, it alerts you that there are unsaved changes and asks you if you want to save them. When you answer yes and hit "save", it repopulates the unsaved POST data back into $input->post before the page editor has had a chance to process it. There are of course some security considerations here, so it has to be built carefully. I should also mention that it won't help much if it's the client's computer or browser that has frozen (there's the PageAutoSave module that can help with that). Though data loss due to a frozen computer/browser is likely even more rare than session loss. I don't have this module fully working just yet (it's a work in progress), but it's relatively simple so it probably won't take long. It's not going to catch everything; it won't save files, for instance. But it will catch the most likely cases, such as changes to those big "body copy" fields that someone might spend hours making edits with. I'll post more about it when I've got it a little further along, if there's interest. Thanks for reading and have a great weekend!

Let me know if you need those changed. https://github.com/search?q=repo%3Aadrianbj%2FTracyDebugger expireNever&type=code

@millipedia I’d love to see your TextformatterLiteVideoEmbed have its own thread in the Forum. Maybe you an kick it off?

I've been testing Hetzner for a couple of months after hearing a lot of good things about them. I've been really impressed so far. Stable, fast, and really cheap for the generous amount of resources (around 8€ for 80 Gigs of space and 8GBs of RAM). Not affiliated with them in any way, just a recommendation on my (admittedly limited) experience with them. I'm using the excellent and free CloudPanel (NGINX control panel) to manage about two dozen low-traffic sites on two different servers with Hetzner in Germany, and I have plenty of resources left in each. I also have a VPS on Vultr managed with the free Cyberpanel control panel, and that's been consistently reliable for a few years, except yesterday when they had a really exceptional downtime of about one hour. I find that Cyberpanel, although easy to use and Apache compatible (easier to deal with for ProcessWire installations), is sometimes buggy as log files sometimes fill up space for no apparent reasons. I've tried a lot of different hosting providers in the last 20 years – from shared to dedicated, managed and unmanaged– but I feel that I have to worry much less about downtimes and unexpected slowdowns with these two.

Very cool! I just found this module and tried the demo. I am amazed how well it already works! Wow!! This could definitely be a game changer for some because it brings so much design flexibility to PW (of course the purists among us cringe inside ?). I have been hoping for some solution like this and will definitely look into it more. Thank you for sharing!

Why don't you just set the field to default to 'today' (in the input tab)?

We look after a lot of non-profit sites including half a dozen schools and they're mostly hosted on Digital Ocean instances managed by Cloudways. Their cheapest instance is currently $11 a month and that's what we normally start and which provides good performance for most cases (there are some additional per Gb backup charges but they're normally very minimal). You can scale up instances if you need to. One thing we find particularly useful is being able to 'clone' sites so we keep a skeleton PW site up to date and can just replicate that to kick off a new project (no, all our sites do NOT look the same...). You can also clone a site to a staging site and then push changes from that to the live site. We generally stick sites behind a free Cloudfare account as well (because honestly why wouldn't you), and Cloudways also have a deal with Cloudflare where you get the enterprise level for $5 a month which seems great if you need it (we rarely do). We also use https://krystal.uk/ for clients who need emails - that's more of a traditional CPanel setup but works well for us.

Hello, Start by checking the ProcessWire logs and then the logs of the webserver/PHP process. You might need to adjust ProcessWire's .htaccess file. Also, more info are needed for us to give you better support. Dev or prod environment, hosting type, version, etc.

It's seem that "OVH/Hosteur/DigitalOcean" VPS are good candidates - I am not affiliated with them, I manage some dedicated bare metal servers and VPS on OVH - you could start with a ~$4 VPS, and upgrade it if you need more vCore and/or RAM without reinstalling the system. By using Ubuntu 23 (v22 support end on July 2023) you can get started in ten minutes if you are used to deploy LAMP/LEMP. On the backend side, you will get a response time of around 60ms, use Cloudflare on front, and your $4 VPS perfs will be really nices. You can check for example the guide on DigitalOcean to choose what suit your needs: concepts/choosing-a-plan

This week I've done some work on the core $cache API (WireCache) to make it able to support other cache storage options. To do that, I had to move all the storage code out of the WireCache class and into a separate class that implements the new WireCacheInterface. So it was kind of a major refactor. Now the WireCache class is independent of storage method, which will make it more flexible in the long term. The first class that implements the new interface is the new WireCacheDatabase. This contains the cache database-storage code that was previously in the WireCache class. But the plan is to also add a WireCacheFilesystem (in progress), and make it possible for others to develop WireCache modules, perhaps for Redis, Memcache, etc. I've been wanting to do this because in some cases I've noticed significantly better read performance from file-based caches. (Though admittedly, at the expense of write performance.) But it made me think it would just be better to have more cache storage options, and also be nice to take advantage of even better cache options available in different environments, like in the AWS environment we run this site on. One of the issues with changing the cache used by the system is that the $modules API (Modules class) depends on WireCache for quite a few things. And the modules basically can't load without the relevant caches being available. At present, $cache has to load before $modules, which makes the whole idea of WireCache-modules a bit of a chicken-or-egg first situation. So I've been working to decouple $modules from WireCache, or at least make it able to function if its cache isn't available on occasion. I made some good progress there, but found that there was a little bit of a performance hit in doing so, so I reverted those changes and put them behind a toggle in the Modules class to experiment with further. But while doing that, I found some other ways to improve the performance of the modules loader. So you'll find the dev branch boots a little faster this week. Maybe not noticeably so (since PW already boots fast), but measurably so. I'm always looking for opportunities to improve performance — even small performance improvements amount to large savings over time. While on the topic of caches, I've also added an experimental $pages->loader()->findCache($selector) method which works exactly like $pages->find($selector) method, except that it caches the page IDs that were found for a period of time that you specify in the 2nd argument (default is 60 seconds). I imagine this method would be useful for complicated or slow page finding operations that don't need to restart from scratch on every request. This is an alternative to markup caching for greater control. But since it caches the result of the find operation (page IDs), and not the actual pages, it has a different set of benefits (and drawbacks) relative to markup caches. I'm still experimenting with this method to get more feedback and make sure it's worthwhile, so far it appears to be. This will likely become accessible at $pages->findCache() once out of the experimental stage. That's all for this week. Thanks for reading and have a great weekend!

Progress is going on well, as usual my choice for Development: TemplateEngineFactory + Twig {% for article in articles %} {% set divisor = 10 %} {% if loop.index0 == 0 %} <div class="flex-center flex-col aa-list"> <h4>Articles</h4> <ul> {% else %} <li> <a href="{{ article.url }}" target="_blank" >{{ article.title }}</a > <span> {{ article.publishedStr }}</span> </li> {% endif %} {% if loop.index % (divisor + 1) == 0 %} </ul> </div> <div class="flex-center flex-col aa-list"> <h4>Articles</h4> <ul> {% elseif loop.last %} </ul> </div> {% endif %} {% endfor %}

Sounds good. Ive found myself implementing a cache in front of $cache that stores stuff in ram (with db persistence too) for data you may need a few times in one request. I realized there was no way to replace $cache with my own extended class, so I just do this in another module. I could probably use a hook as alternative.

Looks like a perfect application for LazyCron. A quick and dirty snippet of code you can place in site/ready.php (or inside a template's PHP file, just make sure that a page with that template is viewed regularly): <?php namespace ProcessWire; // Add daily hook to clean up incomplete, stale pages wire()->addHook('LazyCron::everyDay', null, function(HookEvent $event) { $ts = time() - 86400; // 24 hours ago // Find matching pages, adapt selector however you want (e.g. limit to certain templates or parents) $stalePages = $pages->find("created<$ts, title=''"); foreach($stalePages as $stalePage) { $pages->trash($stalePage); } });

Slightly embarrassed to mention it given the code is currently 90% todo statements, but I did expand this module for a project I did a few months back to include Vimeo. You can grab the code from https://github.com/millipedia/TextformatterLiteVideoEmbed and you can see a test page here https://millipedia.com/video-test/ It definitely needs a lot of tidying up, but I have it running on a couple of live sites at the moment and it hasn't destroyed them yet (ymmv). I'll try and find some time to polish up the code as soon as I can.