PW 3.0.167 – Core updates

[ryan]

ProcessWire 3.0.167 is the newest version on the dev branch and contains the updates mentioned last week, as well as the following: 

Improvements and optimizations to several database fulltext index-based text-searching operators (such as *=, ~=, *+=, ~+=) so that they can better match words too short to index, as well as many stopwords. (code changes)

New $input->queryStringClean() method is like the existing $input->queryString() method except that it enables you to get a cleaned up version according to rules you specify in an $options argument. I added this method primarily because it’s something I need in the core for some planned updates. But it’s in the public API because I thought it might be useful for some other cases as well. 

Kind of technical, but some minor improvements to the $sanitizer array methods were made so that they can now support associative arrays with a ‘keySanitizer’ option that lets you specify which sanitizer method to clean up array indexes/keys.

The $sanitizer->validateFile() method was rewritten and improved plus new ‘dryrun’ and ‘getArray’ options were added.

The core file and image fields have been updated with the ability to require (rather than just utilize) a FileValidatorModule for certain file upload extensions. This was motivated by SVG files being increasingly problematic (or at least my understanding of them) because they can contain the kind of bad stuff that regular markup can (scripts, loading external assets, etc.). But since SVG files are used by many just like the bitmap image formats (JPG, GIF, PNG, etc.) exploits can spread quite easily and unknowingly. I’m not aware of that ever occurring in a ProcessWire site, but it’s one of those things that strikes me as being an inevitable problem for any website (on any software) that has regular SVG uploads. 

ProcessWire doesn’t allow SVG file uploads by default, but you can manually add “SVG” to your allowed upload extensions for any file/image field… and from what I understand now, a lot of people are doing this—using SVG files not just during development, but as an image format that their clients upload too. The usual protections of having a trusted user admin environment don’t help much here. That’s because what matters is not whether the uploading user is trusted or not, but whether the source of the SVG is trusted. And that’s something we have no control over. It seems apparent the core could provide some extra help and guidance in this area. So ProcessWire now requires you install the FileValidatorSvgSanitizer module if you want to have SVG uploads, OR you can check a box in the file/image field settings to whitelist the extension (acknowledging you understand the risks). 

Now you could manually add HTML or JS as allowed upload extensions to a files field as well— and if you are rendering the resulting HTML markup or JS directly in the code of your website, then the same risks would be present as with SVG. But the risks seem obvious for doing this with HTML or JS files. Plus, why would someone do that with HTML or JS files unless for some very specific development purpose? I think it’s unlikely any of us are running websites where clients might be downloading HTML or JS files from other sources and uploading them into the website. And even further unlikely that we’d written the site’s code to include the contents of those files among the site's markup. Whereas, this is essentially what happens with SVG files, from what I understand. As I’ve come to learn this week, it’s even quite common to do this (maybe I’m late to the party). 

Given the above, this week I rebuilt the existing FileValidatorSvgSanitizer module from the ground up, so that it now uses a better/newer SVG sanitization library found by Adrian. I’d suggest installing the module if you are supporting SVG uploads in your site. It seems to do a nice job of cleaning the harmful stuff out of SVG files. I’m not aware of anything it can miss, but I’d still advise some caution with SVG file uploads even if you have the module installed. If you currently allow SVG file uploads but don’t really need them, just remove SVG as an upload option, as that’s still the safest bet. That’s essentially what the current dev branch version does: if you’ve added SVG as an allowed upload extension, it’ll disable it until you decide if you need it; and if you do, then install the sanitizer module or whitelist the extension. 

I know not everyone is going to like that, but it seems like it’s the right and safe thing to do; as well as a good strategy going forward for any other file formats with similar concerns. When our next master version arrives, it’ll be in the upgrade instructions as well. The core has been updated so that it can support the same means for any other file formats we come across in the future that might be problematic in similar ways. 

That’s everything I can remember that’s new to 3.0.167. Thanks for reading and have a great weekend!