ProcessWire 3.0.200 new master/main version

This new main/master version has more than 220 commits, resolves more than 80 issues, adds numerous new features, performance improvements and optimizations, and consumes HALF the disk space of our previous release.

That's right, this is the first major version of PW that is actually smaller—a LOT smaller—and not far in size from the first release of ProcessWire more than a decade ago. This version of ProcessWire also includes numerous optimizations and performance improvements that make it even faster than before. Though these are just two examples of many great improvements you'll find in this new version. This post covers all the details of what’s new in 3.0.200, and there is a lot of great new stuff.

• Contributors and acknowledgements
• Repeater field upgrades
• $pages API additions
• $sanitizer API additions
• $database API additions
• Other API updates and additions
• Admin theme, Inputfield and UI upgrades
• URL-to-page upgrades
• Performance and optimization
• Other additions and improvements
• New hooks
• New modules developed alongside this core version
• Upgrading to ProcessWire 3.0.200

Contributors and acknowledgements

First off, a special thanks to our contributors. ProcessWire 3.0.200 contains new code from at least 10 contributors including (a-z): Adrian, Bernhard, Daun, Horst, Matjazp, MrSnoozles, Radon8472, Robin S., Ryan, TheTuningSpoon and Tyde. Thanks to Teppo at ProcessWire Weekly for his skilled writing and coverage of ProcessWire updates, discussion, modules, sites and more. Thanks Matjazp for recent help in managing the issues repo towards this version, Adrian for helping all of us in our development work with TracyDebugger, Pete for administering our forums, Jan for maintaining the servers, and Robin S. for knowing and solving so many things in this version and others. Thanks also to past contributors, forum moderators, module authors, issue reporters, the ProcessWire community, everyone supporting the ongoing development of ProcessWire, and everyone using it for development with your projects.

Repeater field upgrades

Perhaps some of the biggest new additions to the core were with new features added to repeaters. Most of these new features improve both the core Repeater fields as well as the ProFields Repeater Matrix fields.

Added ability to insert new repeater items before or after existing items (details and video).

Added support for inline cloning of repeater items. This enables you to clone an existing item and have it be placed directly above or below an existing item, according to your preference. (details)

Added option that enables you to specify whether repeater item controls (delete, insert, clone, etc.) should be always visible or visible only when the header is hovered. This can be found on the "Details" tab when editing a repeater/matrix field in: Repeater Editor Settings > When to show repeater item controls/actions.

Core repeater updates also enable new Repeater Matrix options including: matrix type groups, custom type icons, custom add-new options and more. (details)

New option that enables you to specify a background color for Repeater Matrix item labels with a hex color code in your item labels. For instance adding #FF0000 to the repeater item label would make the repeater item header background color red. (details and screenshot)

Repeater and matrix fields can now be configured to use fewer pages. When enabled, it won't create placeholder pages for repeater items until at least one repeater item exists for a given page and field. This can drastically reduce the number of pages consumed by repeaters in your system, and even more so if you are nesting repeaters. (details)

Added the ability to copy and paste repeater items, as well as to clone above or below existing items. It handles this by replacing the existing "clone" icon action with a dialog that now lets you choose among various related actions. Among them is the ability to copy/paste from the same page or between different pages. The only requirement is that the repeater (or matrix) items are from the same field. (video)

Copy and paste in repeaters:

Insert before and after in repeaters:

Custom header/label colors in Repeater Matrix:

$pages API additions

Added $pages->find() support for "id" selectors like id.sort=2|1|3 which ensures the pages are returned in the order given by the IDs. Previously you could only do id=2|1|3 without a guaranteed order.

The $pages->add() method has been updated to pull a fresh copy of the page from the database after it has been added. This helps with some API usages where returned $page may have not been fully initialized in some cases.

Support for OR values on status=value selectors. Now you can match one status or another with selectors by specifying a selector like status=hidden|unpublished, and this will find all pages that either have hidden or unpublished status. The hidden and unpublished values are just likely the most common examples, but you can use any other status name, or as many statuses as necessary in your OR condition.

Support for OR values on sort=value selectors. This one isn't technically an OR condition since we are giving a command to the selector engine about how it should sort, rather than trying to match something. You can tell it how to sort with the syntax sort=date|title as an example. That would be shorthand for sort=date, sort=title, which is saying "first sort by date, then by title".

Support for combined start and limit selectors. Previously you have had to specify start=x, limit=y separately, if you needed it. Now you can optionally specify both as part of the limit, for example limit=5|10 is shorthand for start=5, limit=10, and actually kind of similar to what it translates to in MySQL, which is "LIMIT 5,10". I suspect that not many people never actually use "start=" in their selectors unless using "start=0" to prevent a set from paginating. So if you wanted a set of 10 pages that don't follow the current pagination (and start from 0), you could specify limit=0|10 in your selector, or the more verbose start=0, limit=10 will continue to work in any PW version.

Support for matching children paths. Previously you couldn't do children=/path/to/page to match a page having the given child path. Though you could do children=123 where 123 is the ID of the child page. It has been updated so that it can now support paths in addition to IDs, just as "parent" selectors do. You can use OR values here too if you'd like.

New $pages->request()->getClosestPage() method to identify the closest matching page when a 404 occurs. (details)

Added a new $pages->new() method. This method is similar to the $pages->add() method, in that it is used to add new pages to the database, but it accepts a selector argument, making it more flexible and easier to use in many cases. The $pages->add() method has arguments that you have to remember, whereas $pages->new() accepts a single-argument selector string, like many of the other $pages API methods. This new method can also auto-detect some properties that the add() method cannot (like parent or template). (details).

Upgrades for $pages->findRaw() including: Support for selecting parent properties/fields in the return value; Support for selecting page 'meta' data in the return value; Support for selecting "references" in the return value (references are other pages that reference the found pages via Page reference fields); Support for selecting title and/or value from options (FieldtypeOptions) fields; Support for a new "flat" option that flattens multidimensional arrays in return value. (details)

The $pages->findRaw() also now has a new 'nulls' option that will add placeholders in the return value with null values for any fields you requested that were not present on each matching page. Without the nulls option (the existing behavior) it retains the more compact return values, which omit non present values completely. (details)

Added new $pages->loader()->findByName() method which is a method optimized just for finding pages by name as quickly as possible, without performing any filtering or access checking.

Added new $pages->getInfoByPath() method which is a shortcut to the new PagesPathFinder get() method. It returns a verbose array of info about a given page path. See linked method for details.

Added new $pages->raw()->col($id, $col) which returns the raw value for a requested native pages table column ($col) for a given page $id. Also added $pages->raw()->cols($id, $cols) which does the same thing but returns multiple column values in an array.

$sanitizer API additions

New $sanitizer->words() method reduces a string to contain just words without punctuation and other sentence structure.

Add $sanitizer->entitiesA() method that works the same as the entities() method except that it will also traverse arrays (recursively) to entity encodes values within them. Also added the $sanitizer->entitiesA1() method that does the same but only for values not already entity encoded.

Add trim option to $sanitizer->array() method which trims string values in an array of leading/trailing whitespace. Default value for this option is true.

The $sanitizer->float() method (as well as FieldtypeFloat and InputfieldFloat modules) have been upgraded to support E notation (i.e. 1E-2, 10E-14, -1.253E-5, etc.)

Improvements to $sanitizer->getTextTools()->markupToText() method with new clearTags and linksToMarkdown options. The clearTags option lets you specify which tags should have content cleared rather tahn converted to text (default is script, style and object tags). The linksToMarkdown option lets you convert links to [text](url) rather than removing.

$database API additions

Add $database->renameColumns() and $database->renameColumn() methods that enable you to rename columns without having to know the column type (something that MySQL versions prior to 8.x require).

Add a $getNumberOnly argument to $database->getVersion() method to return the database version number without vendor suffix.

Add a $database->getServerType() method that returns the MySQL server type: MariaDB, MySQL, Percona, etc.

Add new verbose=3 option to $database->getColumns() method to get the column type as used in a CREATE TABLE statement.

Other API updates and additions

Useful new $files->getCSV() and $files->getAllCSV() methods which abstract away the redundant details of reading a CSV file into one simple method call. (details)

New $user->setLanguage() method for quietly setting a user language at runtime.

Upgrades to InputfieldWrapper class including new insert() and getByAttr() methods, plus expand the insertBefore() and insertAfter() methods to also accept Inputfield names or arrays.

WireHttp was updated to support sending cookies in GET/POST requests powered by CURL via the new $http->setCookie() method.

Improvements to the WireHttp::send() method and updates to use CURL first when available (rather than fopen).

New wireLength() function which duplicates the behavior of mb_strlen() in PHP prior to version 8.1. A non-multi-byte wireLen() function was also added which duplicates the behavior of strlen() in PHP versions prior to 8.1.

Update WireArray::sort() to sort more like PHP's sort so that null or blank string values are sorted rather than appended. Added a SORT_APPEND_NULLS flag for the previous behavior too (see sort flags).

New $fields->findByType() returns all fields using given type without triggering all to lazy-load, with options to control what kind of return values you want and ability to find fields inheriting from types.

New $fields->getAllNames() returns array of all field names without triggering all fields to lazy-load.

New $datetime->strftime() method that duplicates much of the behavior of PHP's strftime() function in versions of PHP that have dropped it (PHP 8.1+).

Admin theme, Inputfield and UI upgrades

New addition by Bernhard that lets you provide your own custom render files for different parts of the Uikit admin theme by placing them in /site/templates/AdminThemeUikit/. Third party modules can also define custom render files. Furthermore, this addition adds a new hook that enables you (or your modules) to hook into and modify just about any part of the admin theme rendering. (details)

Upgrades to the InputfieldPageListSelectMultiple module by way of Robin S. This enhancement keeps the selectable page list open for multiple selections rather than closing it for each selection. It also better indicates when an item is selected. GIF screencast further down. (details)

New collapsedNoLocked "visibility" option for Inputfields which is labeled "Open + Locked (not editable)". (Inputfield collapsed constants)

Dependent select fields can now be used inside of Repeater items. To work in a repeater, your Page fields configured with a selector string that refer to page.property should instead refer to item.property. Outside of a repeater item, the "item.property" behaves the same as "page.property".

Updated ProcessWire's page editor (ProcessPageEdit) so that the "Template" selection input (on settings tab) is Ajax-loaded, like the existing "Parent" selection input.

UI improvement that prevents the AsmSelect inputs (used throughout PW) from marking an item "pending delete" (strikethrough), unless the item was already selected when the page rendered. If the item was selected and then removed prior to save, it will simply disappear from the selection rather than showing a "pending delete" state. This prevents unnecessary "are you sure" confirmation questions following save.

Hovering module titles in the Modules list now shows the module class names in a tooltip. (via Bernhard)

Below: upgrades to InputfieldPageListSelectMultiple module via Robin S.

URL-to-page upgrades

ProcessWire's entire URL-to-page system was rewritten to be a lot more efficient, faster and more maintainable.

A new PagesRequest class has been added and it can be accessed from $pages->request(). Its primary focus is on one hookable method: $pages->request()->getPage(). The method analyzes the current request, identifies the page to render, and returns it. Previously this logic was in the ProcessPageView module. The benefit of this method is that you can now determine the page to render much earlier, like during the boot process. Previously the current page wasn't identified until after PW booted and the ProcessPageView module loaded. This will enable [for example] a module such as SessionAllow to decide whether or not to allow sessions based on what page was requested. In addition, this method is hookable, so you could have a module that adds its own logic to identify or change the page that should be rendered.

A new PagesPathFinder class was added which can be accessed from $pages->pathFinder(). Like the PagesRequest class, this new class primarily focuses on one method: $pages->pathFinder()->get($path). The given $path can contain any page path, optionally containing language prefix, URL segments, pagination numbers, or even a previously named version of the path. It will find and validate that path and return an array of information about it. PagesPathFinder does its job very quickly. In fact, it's about 10 times faster (in testing/timing locally) than PW could previously identify this information using existing methods. PagesPathFinder will also take advantage of the PagePaths and/or PagePathHistory modules when appropriate, if they are installed. (details)

The core PagePaths module has been refactored and now has multi-language support. Previously it only worked on single-language sites. Now multi-language sites can take advantage of potentially significant performance improvements offered by this module. This is one module that is not installed by default, so you may want to consider installing it: Modules > Core > PagePaths.

The core PagePathHistory module has been refactored. While built primarily for internal use, it now has a powerful new getPathInfo() method that gets array of info about a path, when it has historical information.

Performance and optimization

Smaller and faster is a major and recurring theme you'll find in ProcessWire 3.0.200.

The size of the core has been cut in half, from 15.5 MB down to 7.5 MB. This was done by removing all the site profiles except for the site-blank profile. This brings the core back near the size that it was more than a decade ago with ProcessWire 2.1. (details)

Field and template performance and scalability improvements were made with lazy loading fields, templates and fieldgroups. This comes via TheTuningSpoon and was actually one of the most major updates to this version of ProcessWire. Previously ProcessWire loaded all fields, templates and fieldgroups into memory on every request. If there were a lot of them (hundreds to thousands) then it could slow the boot time. Now ProcessWire lazy-loads these resources, loading only those that are needed for the request. This results in a significant performance improvement and reduced memory footprint. Though you'll have to have a large number of fields or templates before you'll necessarily feel the difference. But even when Lazy Loading is disabled, this version manages templates, fields and fieldgroups about twice as fast as before. (details)

Most of ProcessWire's site profiles have been moved into their own dedicated repositories.

The URL-to-page system in ProcessWire (mentioned earlier) was completely rewritten for significantly increased performance.

The module that manages multi-language URLs in ProcessWire (LanguageSupportPageNames) has been refactored for increased performance. You shouldn't notice any difference other than that it's faster. For instance, one of the changes is that it drops its old indexes on the pages table and re-creates them in a different way for faster queries.

Significant performance improvements have been made for the $pages->findMany() method. It now uses temporary caches to avoid re-loading supporting pages common among matched pages in each returned set. How does it do that? Read the next item about "runtime page cache groups" to find out. (details)

Added support for runtime page cache groups. This enables pages to be cached as a group, or more importantly, uncached as a group. It was added primarily to add efficiency to $pages->findMany(), so that it can cache supporting pages (like parents of pages returned by findMany). Previously, it would have to load a fresh copy of each supporting page used by findMany(), for every returned page, since findMany() used no in-memory caching (otherwise you could run out of memory on large results). So if you iterated a $pages->findMany() result and output the URL of each page (which triggers parents to load), then it would have to reload all those parents for each iteration. Now it can cache them for each chunk of 250 pages, offering a significant potential performance improvement in many cases. But this update isn't exclusive to findMany(), it can be accessed from the public API via $pages->cacher()->cacheGroup() and $pages->cacher()->uncacheGroup() should anyone find it useful for other purposes.

Other additions and improvements

Adding a field to a template using the "Send to template(s)" option in the field editor now updates the Template's modified date.

Support has been added for searching subfields with PageAutocomplete using the field.subfield syntax.

ProcessWire's installer has received several updates, most related to instruction for site profiles no longer included in the core.

ProcessWire and the Profile Exporter module now support a custom /site/install/finish.php file which is a template file that is called when installation of a new ProcessWire and site profile have finished, but before it has cleaned up the installer. (details)

A new "advanced mode" feature that lets you edit the raw configuration data for a module has been added. This can be useful for various reasons, especially for module developers. If you have $config->advanced = true; in your /site/config.php file, you'll see a new option on your module information screen that enables you to directly edit the raw JSON configuration data for the module. (details)

System update 20 was added and it fixes the "created user" for several system pages. Previously these system pages had the "id" of a non-user page, which didn't make sense.

CKEditor has been updated from 4.16.0 to 4.18.0. See the CKEditor release notes for details which also include security fixes.

The multi-language URLs module (LanguageSupportPageNames) gained a new configuration setting that lets you specify what should happen when a page is accessed at a language it's not available in. The options are: "throw a 404" or "redirect to the default language".

ProcessWire's PageFinder was updated to allow partial text matching in default language page paths in multi-language environment.

New hooks

Fieldgroups::fieldRemoved($fieldgroup, $field)
Called after a field has been removed from a fieldgroup/template.

Fieldgroups::fieldAdded($fieldgroup, $field)
Called after a new field has been added to a fieldgroup/template.

Fieldgroups::renameReady($template, $oldName, $newName)
Called before a fieldgroup is about to be renamed.

Fieldgroups::renamed($template, $oldName, $newName)
Called after a fieldgroup has been renamed.

Templates::renameReady($template, $oldName, $newName)
Called before a template is about to be renamed.

Templates::renamed($template, $oldName, $newName)
Called after a template has been renamed.

Fields::renameReady($field, $oldName, $newName)
Called before a field is about to be renamed.

Fields::renamed($field, $oldName, $newName)
Called after a field has been renamed

ProcessPageEdit::ajaxSaveDone(Page $page, array $data)
Makes it possible to control the JSON output returned after an ajax save operation

ProcessLanguage::processCSV($csvFile, Language $language, array $options = [])
Process a CSV file to import changes from it (made hookable per request).

New modules developed alongside this core version

New “PageAutosave + LivePreview” module was developed for this new core version and is available for download now in ProDevTools and ProDrafts. This module auto-saves your changes in the page editor and also lets you preview those changes in a separate window, all as you type (details and more details).

New FormAutoSaver + Reminder module is now available in ProDevTools and FormBuilder. The module automatically saves front-end forms before they are submitted. This ensures users cannot lose their entered form data, whether they forget to submit the form, their browser freezes, they take a break from the form, or even if they switch devices. It sends the user one or two reminder emails to complete their form. Works with both FormBuilder forms and your own independently developed forms.

Upgrading to ProcessWire 3.0.200

Like most ProcessWire upgrades, so long as you are already running some 3.x version, you should be able to upgrade just by just replacing the /wire/ directory, though I recommend these steps to do that:

1. Backup your current site database and files.
2. Edit your /site/config.php and set $config->debug = true;
3. Login to your admin and keep a browser window open to it.
4. Replace your /wire/ directory with a copy of it from the new version.
5. Do a “reload/refresh” in your browser until all upgrade-related notifications clear.
6. Test your admin and front-end site thoroughly to make sure everything works.
7. Edit your /site/config.php and set $config->debug = false;

This is known as the general upgrade process. Always test upgrades in a staging or development environment before migrating to a production environment.

If you are currently running a version of ProcessWire prior to 3.0.135 then you'll also want to upgrade your .htaccess file.

Still running ProcessWire 2.x? See upgrading from ProcessWire 2.x.

Upgrade note

One significant change in 3.0.200 (relative to the previous master) is that ProcessWire now uses lazy-loading fields, templates and fieldgroups. This is a fairly major system-level change, so be sure to test thoroughly after upgrade. If you notice any issues, you may want to try disabling lazy-loading by editing your /site/config.php file and adding:

$config->useLazyLoading = false; 

Please let us know if you run into any issues by opening an issue report or posting in the forum.

Upgrading your 3rd party and Pro modules

After a core upgrade, it's also a good time upgrade your installed modules as well. I recommend installing the ProcessWire Upgrade module which will check the versions of all your installed modules at once, and let you know of any that have upgrades available. Newer versions of this upgrade module are also aware of all current Pro module versions as well, should you have any installed. If you want to upgrade any Pro modules and no longer have active support/upgrades then feel free to contact us and we can always reset the renewal option.

Troubleshooting

In addition to the upgrade note above, we also have a troubleshooting upgrades guide in case you run into any errors during upgrade.

Comments

• 

  Lolo Ogly

  • 3 years ago
  • 61

  Great job! Thank you Ryan and company for ProcessWire.

  RestAPI/GraphQL and SQLite support out-of-the-box, please?

  Reply

• 

  Donatas

  • 3 years ago
  • 21

  Great news and very welcome updates! Thank you Ryan and all the contributors!

  Reply