ryan

@wbmnfktr ProcessWire 3.0.218 moved its cache storage from $cache to the modules table, so those module names you see there are actually the cache names from 3.0.218+. This error would be expected if downgrading from 3.0.218 or 3.0.219 to 3.0.217 or prior, since prior versions don't know about those caches in the modules table. It's not a fatal error and it's okay to delete them from the modules table if you downgrade to a previous version. (or it's okay to ignore them). But this error wouldn't be expected if upgrading to 3.0.219. Your message suggests this is what you saw after UPgrading, but this is what I'd expect to see if DOWNgrading, so just wanted to confirm?

If you upgrade to 3.0.219, grab the latest commits, as the commit labeled "Bump version to 3.0.219" still had a couple of issues that I ran across since then and have fixed. (The issue may have originated in 3.0.218 or 3.0.217 actually). It was primarily "class not found" errors, and I didn't notice at first because the errors didn't appear when doing minor version upgrades and instead would appear when upgrading older versions to 3.0.219.

@thetuningspoon @cb2004 Try the most recent dev branch commits. The "class not found" error at least was coming from upgrading an older version of PW that had certain modules in different parent directories. For instance, /wire/Fieldtype/FieldtypeFile.module was moved to /wire/Fieldtype/FieldtypeFile/FieldtypeFile.module many versions back. But if upgrading from that older version (that had the file in a different location) to 3.0.218 then you'd see this error. At least, that's the issue I ran into (and fixed) here.

ProcessWire 3.0.219 on the dev branch focuses primarily on a major overhaul to the core $modules system. The Modules class had grown into more than 5000 lines of code — all related to modules, but with lots of different areas of focus within that. It had become a little bit messy, fragile and hard to maintain at that size. I've had @todo notes in the class to "clean it up" for quite awhile, but this week I finally got to it. The approach is similar to that of our $pages API (Pages class), which is split into separate classes for page finding, loading, editing, caching, etc. The Modules class has been split into several much more focused classes for module information, installers, loaders, files, flags, duplicates, and configurations. This leaves the main Modules class as the gatekeeper, making it less fragile and easier to maintain. If this overhaul was perfect, you shouldn't notice any difference, and the public $modules API remains as before. But with such significant overhaul that took a full week to complete, there's also an increased potential for temporary errors, so please let me know if you encounter any. Thanks for reading and have a great weekend!

@MarkE What I'm intending to communicate is that it's up to the module developer how many of the WireCache features they want to support. That's because, as I understand it, it's not possible to support the the full WireCache feature set with something like Redis. The core WireCache (WireCacheDatabase) will continue to support the full feature set, as it always has. For 3rd party WireCache modules, the required feature set would just be the ability to get, save and delete cache by name ... what the majority of cache use cases are. If they can support more than that, great. But I think it's better to keep the door option to more cache options by not requiring specific and lesser-used cache features unique to PW. If a 3rd party module has to support the full WireCache feature set then likely there won't be 3rd party WireCache modules. For cases like yours, where you are depending on specific features, you'd probably want to stick with using the core default (WireCacheDatabase), unless future 3rd WireCache module is able to support them. @BitPoet Great! Yes, that's correct, the core does not use anything but get/save/delete by name. But it does use wildcard matching (for FileCompiler). Does Redis allow for matching by partial cache name? For instance, a cache name that has a wildcard at the end like "MyCache*"? If not, there are ways around it, but I was just curious. Maybe that is a good way around it. Though having to hit the database in addition to Redis might reduce the benefit, or make it slower than just using the database cache. Maybe another way around it is for WireCache to fallback to the WireCacheDatabase when a particular feature is needed that's not supported by the WireCache module. Good idea, I will add that. @kongondo I don't think so because the cache would have to write and read from the same place, otherwise I don't know how it would find caches that it wrote. But PW does support separate read and write connections for the database (WireCacheDatabase), and it will use them when they are provided.

@adrian I don't think there's any need to change it, as cache handlers that don't support past dates can substitute a future date, i.e. if($expires === WireCache::expireNever) { $expires = wireDate(WireCache::dateFormat, '+1 YEAR'); }

@BitPoet This is for the maintenance processes. WireCache performs maintenance every 10 mins and whenever any page or template is saved. The page/template save maintenance only happens if rows have expiry dates prior to WireCache::expireNever. Finding by expiry dates can be optional if the cache (Redis) handles expiration on its own. I'm also not sure how much need for these two constants anymore since the modules are now decoupled from the cache. Though I think TracyDebugger uses expireNever. You can just replace WireCache::expireNever with some date far in the future (a year?). The WireCache::expireReserved can be treated the same as expireNever. It was primarily for the modules-system and I will likely deprecate this option since it's no longer needed. Its only purpose is to behave like expireNever but also prevent the rows from being deleted when a "delete all" action is being performed. We don't need that anymore. There are a few expirations prior to expireNever that could also considered. Either that, or the WireCacheInterface module could just indicate it doesn't support them. Since these are all dates before WireCache::expireNever, they can be easily filtered in the same way. If the cache can't search by expiry, then implementing them would mean mapping to some other property that is searchable by the cache handler. Any WireCacheInterface module doesn't need to know about any of these since all the logic for finding them is mapped to expiration dates: WireCache::expireSave - This expiration means that the cache row should be deleted whenever any Page or Template is saved. WireCache::expireSelector - This is like expireSave above except that it indicates the cache value is JSON-encoded along with a property named 'selector' and this contains a page-finding selector. WireCache loads them all and when the saved page matches any selector then the cache row is deleted. WireCache maps template IDs to dates by doing $expire=date('Y-m-d H:i:s', $template->id); and that can be reversed with $templateID=strtotime($expire); When a page having that template ID is saved, the cache row having that templateID-based expire date is deleted. Like with the above, the WireCacheInterface module doesn't need to know anything about this other than finding rows having those expiration dates, and presumably those could just be mapped to something else. I'm not sure it's necessary to support these other than mapping expireNever to a future date, as I think very few actually use these features. From what you've described, Redis handles a lot of the stuff that WireCache does on its own. If the cache handler (Redis) can handle these tasks on its own, that's great. If I'm understanding what you've described correctly, here's how I would handle it: When find($options) receives a request containing $options['names'] to find, then ignore anything in the $options['expires'], since Redis is never going to return an expired row. When find($options) receives a request with empty $options['names'], and a populated $options['expires'], just return a blank array (no results). When save() receives an $expire date of expireNever or expireReserved, substitute some far future date instead. When save() receives an $expire date prior to expireReserved, throw an Exception to say the feature isn't supported, or substitute 1 hour, 1 day, or whatever you think is appropriate. Once it works with those, if you wanted to support the full feature set, then you could always go back and see about mapping expiration dates less than or equal to WireCache::expireNever to some other searchable property (and using a far future expiration date). But very few people actually use the page/template/selector clearing features (as far as I know), and the core doesn't need them, so they could be considered optional.

Continuing from last week's post and discussion, ProcessWire 3.0.218 decouples the modules system from the cache system. Now the modules system maintains its own internal caches (at least once you do a Modules > Refresh). It'll still use the $cache API as a backup (temporarily), but now you can safely export the database without the "caches" table, or even delete the "caches" table, if you want to. It'll get re-created as needed. In this version, work also continued on the new WireCacheInterface (and major updates in WireCache) so that we could support external modules to handle cache storage. This capability is kind of similar to how we support 3rd party WireMail and WireSessionHandler modules. The first example is WireCacheDatabase, which is the default cache storage handler for the core. And today we have a new module called WireCacheFilesystem that replaces the default WireCache database storage with a file-system based storage, once installed. It's not yet clear if there are major benefits one way or the other (cache in database vs. file system), as I've not been able to put all this new code through performance testing yet. I'd definitely be interested to hear if anyone has a chance to test things out. I expect the file system might be faster for reading caches, while the database may be faster for writing caches. At least that's what I found with a few preliminary experiments, but they haven't been very thorough, so take that with a grain of salt. I thought we needed at least 2 examples of classes implementing WireCacheInterface before we'd be ready to support potential 3rd party WireCache modules. I imagine that 3rd party modules getting into dedicated cache options independent of database or file system is where we'll start to see major performance benefits. At least for sites that use the cache heavily. That's all for this week, have a great weekend!

@bernhard Usually ProfilerPro would be great for this, but in some cases I can't use it because ProfilerPro is a module and I'm timing something that happens prior to modules loading. When PW is in debug mode, it times all of the boot processes, so I can just look at those. I also add other Debug::timer() calls as needed to time specific things. When testing in the admin, you can finish a timer with Debug::saveTimer($timer); and it'll show the result in the Debug > Timers section of the admin. But you can't look at any one instance of a timer. You've got to take several samples before you really know if there's a trend in the numbers. I'm usually looking for consistently 10 ms or more when it comes to improvements. @Ivan Gretsky The cache supports an expiration flag represented by the WireCache::expireReserved constant. It means that rows with that flag are reserved and should never be deleted. If you use the $cache API to clear the cache, it'll do it in a safe way. But if you just omit the caches table from a DB dump or manually delete all the rows in the caches table, then it would be problematic. I agree it would be better not to have to consider this. I'm not sure we need to keep that flag other than for modules, so this is one reason why I'm looking to have the modules use some other storage method for its cache. Though if new caching options came along (Redis, etc.) it would sure be nice for modules to be able to utilize that caching method too, so there are tradeoffs. Ideally, the modules would still use WireCache, but be able to recover easily if its caches get deleted, so that's probably what I'm going to work towards, but it's not as easy as it sounds. @teppo Sounds great! I have no experience with Redis but this server seems to have it and I've also been curious about it. I really like the idea of dedicated cache systems independent of the DB or file system. I'd definitely like for PW to be able to support them, and am glad to hear you are interested in it too.

This week I've done some work on the core $cache API (WireCache) to make it able to support other cache storage options. To do that, I had to move all the storage code out of the WireCache class and into a separate class that implements the new WireCacheInterface. So it was kind of a major refactor. Now the WireCache class is independent of storage method, which will make it more flexible in the long term. The first class that implements the new interface is the new WireCacheDatabase. This contains the cache database-storage code that was previously in the WireCache class. But the plan is to also add a WireCacheFilesystem (in progress), and make it possible for others to develop WireCache modules, perhaps for Redis, Memcache, etc. I've been wanting to do this because in some cases I've noticed significantly better read performance from file-based caches. (Though admittedly, at the expense of write performance.) But it made me think it would just be better to have more cache storage options, and also be nice to take advantage of even better cache options available in different environments, like in the AWS environment we run this site on. One of the issues with changing the cache used by the system is that the $modules API (Modules class) depends on WireCache for quite a few things. And the modules basically can't load without the relevant caches being available. At present, $cache has to load before $modules, which makes the whole idea of WireCache-modules a bit of a chicken-or-egg first situation. So I've been working to decouple $modules from WireCache, or at least make it able to function if its cache isn't available on occasion. I made some good progress there, but found that there was a little bit of a performance hit in doing so, so I reverted those changes and put them behind a toggle in the Modules class to experiment with further. But while doing that, I found some other ways to improve the performance of the modules loader. So you'll find the dev branch boots a little faster this week. Maybe not noticeably so (since PW already boots fast), but measurably so. I'm always looking for opportunities to improve performance — even small performance improvements amount to large savings over time. While on the topic of caches, I've also added an experimental $pages->loader()->findCache($selector) method which works exactly like $pages->find($selector) method, except that it caches the page IDs that were found for a period of time that you specify in the 2nd argument (default is 60 seconds). I imagine this method would be useful for complicated or slow page finding operations that don't need to restart from scratch on every request. This is an alternative to markup caching for greater control. But since it caches the result of the find operation (page IDs), and not the actual pages, it has a different set of benefits (and drawbacks) relative to markup caches. I'm still experimenting with this method to get more feedback and make sure it's worthwhile, so far it appears to be. This will likely become accessible at $pages->findCache() once out of the experimental stage. That's all for this week. Thanks for reading and have a great weekend!

@szabesz The roles/users/templates part would be possible. But for someone that has access to the page editor, is there any reason not to give them access to save/restore something that they lost? The tool doesn't give them the ability to restore any fields beyond what they already have access to. I guess I see it as something that ideally wouldn't be limited by access control since it's a tool to prevent lost work, rather than a tool providing any kind of enhanced access. Though maybe there are use cases I haven't considered yet. As for deciding what fields should be restorable, that part is a little more tricky because it doesn't actually have any idea what fields it's saving or restoring. It's not actually saving or restoring fields. Instead, it's saving and restoring the raw POST data from the request. It would be easy to identify a TinyMCE field named "body" in the POST, but it would be quite a bit more difficult to identify the the same field within a repeater or other more complex type. If it offered the ability to select which fields it can save/restore, then it would have to know the POST data naming conventions used by all of the fields and subfields within them. This is where it would get significantly more complicated to configure and maintain, and more prone to bugs. Whereas now it just simply restores your lost POST request, providing the same result as if your "Save" had worked, rather than failing due to session loss. It doesn't have to know what all the variables in the POST request are for, so by just saving/restoring it all, there's very little chance for the module to screw up anything.

Following up from the module mentioned last week, the PageEditRestore module has been released and here's a new blog post with all the details. This module helps to prevent page edits in the admin from getting lost when the user’s session is lost— https://processwire.com/blog/posts/page-edit-restore-module/

Support thread for the PageEditRestore module. This module helps to prevent page edits in the admin from getting lost when the user’s session is lost. Blog post about Page Edit Restore Module directory page GitHub page

@ttttim Is it possible that new version of Safari is swapping around stuff in the useragent string? You might try setting $config->sessionFingerprint = false; temporarily in your /site/config.php file, just to see if that's the source of it. I would also try logging in from a new "Private" window in Safari, to see if might be cache/cookies like Bernhard mentioned.

@bernhard The Pastefilter may be good for this specific case, since you are wanting to filter pasted content. But also have a look at the valid_elements and/or invalid_elements options, which give you control over what tags and attributes are supported in the input.

This week ProcessWire 3.0.217 is released with 10 issue fixes, 2 PRs and a couple of minor additions too. See the dev branch changelog for details. Recently a client called me in a panic because they'd spent a few hours making edits to a page, and when they finally hit save, they were no longer logged in, so their changes were seemingly lost. I guess that their IP had changed somehow, or they kept the page editor open overnight or something. Whatever it was, they were now sitting at the login screen with their changes apparently lost forever. Luckily this person left that window as-is and contacted me to see if there was any way I could recover their changes. I quickly edited their /site/config.php file and temporarily added these: $config->protectCSRF = false; $config->sessionFingerprint = false; Next, I asked them to open another tab and login there. Once logged in, they returned to the tab where the page save failed, then hit "reload" in their browser, and their changes were saved. Phew. Thankfully that worked, but if it didn't, the next thing we were going to try was to open the browser inspector "Network" tab, and then copy/paste the edited content right out of the browser's POST data and into the CKEditor HTML source window. I imagine this has happened to others and perhaps they weren't so lucky as to recover the unsaved changes. So how can you avoid this issue? The best bet is to just save your work regularly. But that doesn't always happen, no matter how many times we communicate that to the client. So you can reduce the probability of it by making a couple adjustments to your config.php file. One change would be increasing your $config->sessionExpireSeconds. But the default is already 86400 seconds (1 day), and I'm not sure many really take more than a day between starting an edit and saving it... though I'm sure it happens. Another change would be turning off the $config->sessionFingerprint (or loosening it, see fingerprint settings). That's trading security for convenience, which isn't ideal, but it would prevent a changed IP address from expiring the session. Another thing you can do is install the ProDevTools UserActivity module, which keeps a ping going to the server, preventing you from getting logged out due to inactivity. Though this doesn't prevent a changed session fingerprint from logging you out, though it at least alerts you as soon as you've been logged out. Even the above changes might not completely solve this issue, and I don't like to tailor session settings around this case either (reducing security), so I've been thinking of alternatives. After dwelling on it for awhile, I started working on a module that saves non-authenticated POST requests sent to the page editor... saving data that would otherwise get lost. Then when you go back to edit the page, it alerts you that there are unsaved changes and asks you if you want to save them. When you answer yes and hit "save", it repopulates the unsaved POST data back into $input->post before the page editor has had a chance to process it. There are of course some security considerations here, so it has to be built carefully. I should also mention that it won't help much if it's the client's computer or browser that has frozen (there's the PageAutoSave module that can help with that). Though data loss due to a frozen computer/browser is likely even more rare than session loss. I don't have this module fully working just yet (it's a work in progress), but it's relatively simple so it probably won't take long. It's not going to catch everything; it won't save files, for instance. But it will catch the most likely cases, such as changes to those big "body copy" fields that someone might spend hours making edits with. I'll post more about it when I've got it a little further along, if there's interest. Thanks for reading and have a great weekend!

300-600kb are pretty large chunks. I would go with files to keep the DB size down.

This week updates continued with various .js files in the core (for jQuery 3.6+ compatibility). A few external libraries were also updated to the latest versions, including jquery-tablesorter, jquery-ui-timepicker, magnific-popup and vex. And finally, InpufieldTinyMCE has been added to the core! That's a lot of updates, but there's not a lot more to write about these updates because we covered the jQuery updates last week, and InputfieldTinyMCE has been the topic of several past posts. But now we've got everything together and running smoothly in ProcessWire 3.0.216. I think we're now ready to focus on getting the next main/master version out in the coming weeks. There likely won't be an update next week because I'll be traveling for a few days, but will be back to a regular schedule the following week. Thanks for reading and have a great weekend!

@matjazp Thanks, I have updated all of those and others along the way. The only one I didn't update was Selectize because we're using what's called the "standalone" version, and I don't see it mentioned at all in the current version, so seems like there's something I'm missing and figure it's better to leave that one for now. (Especially if the new version isn't yet updated for jQuery 3.x.) Regarding Magnific, I had a look at the mentioned security leak and I don't think it's an actual security issue, if I understand correctly. It looks like it would require someone able to manipulate the image filename to insert XSS into the actual filename. If someone can do that then the installation would already be compromised whether Magnific is there or not.

I'm not yet sure what ProcessWire could do here since it's the template file that controls all the logic of what gets output. But I may not yet fully understand the request, so I'll use an example or what I do understand below. Markup Regions don't have control over what your template file spends time rendering, just what gets output at the end. So there wouldn't be much benefit to having output of partials when it still has to spend the time to render everything, whether used in the output or not. Instead, you would need some logic in your template file in order to selectively render partials, and gain a performance benefit from it: <?php namespace ProcessWire; // render just $part if requested, otherwise render all parts $part = $input->get('part'); // i.e. header, content, footer ?> if($part == 'header' || !$part): ?> <div id='header'> ...header markup... </div> <?php endif; ?> if($part == 'content' || !$part): ?> <div id='content'> ...content markup... </div> <?php endif; ?> if($part == 'footer' || !$part): ?> <div id='footer'> ...footer markup... </div> <?php endif; ?> <?php if($part) return $this->halt(); ?> In the above example, if the page is requested without a "?part=" query string in the URL, then it renders everything (header, content and footer). But if rendered with a "?part=content" query string in the request URL (for example), then it would render and output just the <div id='content'>...</div>.

@matjazpOne thing that's not totally clear to me is if the jQuery focus() and blur() methods are actually deprecated? I'm probably missing something, but so far I can't find anything on the jQuery site that indicates those two are deprecated. It would make sense that they would be since many of the other shorthand methods are.

@matjazp Thanks!! I will update those files and the jQuery TableSorter version to the one you linked.

@artfulrobot There's different ways you could go about it, but what you described should work. The way I built the comments form (here) for the example I linked earlier was to use a FormBuilder form for the comments/reviews form and just used the comments API to add comments. (In that example the "Rate more details" link at the bottom opens a bunch more fields). I mainly used FormBuilder because there were so many different fields and photo uploads, etc., that it went beyond what I wanted to do with extending a CommentForm class. Though a manually written regular HTML form would have also worked fine. I use CommentForm more often when it's more typical blog-type comments with the built-in optional stars, votes, etc., as it can save a lot of time since it's nearly turn-key. If you only need to add a field or two, that's probably the quickest route. Btw, I also see no harm with using pages for comments either, but just that you'd be building a lot from scratch with regards to spam prevention, comment approval, etc., and also just as a personal preference, I like to keep anonymous user generated content out of the page tree.

@artfulrobot Comments are a kind of turn-key fieldtype focused just on comments (like you might use in a blog) or reviews, and their purpose is pretty specific and different from that of a page. So the point is more to be focused on solving a specific thing than to be flexible in the way that pages are. And actually, this is the purpose of most Fieldtypes. If what you are needing is the ability to build your own custom type then that's what pages, templates and fields are for, and maybe that's what you need, I'm not sure. But if you are needing specifically comments, then FieldtypeComments is also quite flexible for comment-specific needs. When it comes to custom data that you want to store along with the comment, there is the meta() method which you might find useful: https://processwire.com/api/ref/comment/meta/. This is what I use for storing photos and other Q&A with comments/reviews, like you see here: https://www.biketours.com/reviews/

Thanks, I pushed the fix from matjazp