Weekly update – 8 November 2024

[ryan]

This week on the dev branch, we have a new $page->preload() method that enables you to preload multiple fields in one query (in this GitHub commit). This is kind of like what the autojoin option does (when configured with a field), or what the $pages->findJoin() method does, but with one big difference. Those options happen as part of the page loading process. Whereas $page->preload() can be applied to a page that has already loaded. 

Here’s one example where you might find this useful. Say you have a page living at /products/product/ and it has a hundred fields. At the top of your template file that renders the page, you could have a $page->preload(‘field1’, ‘field2’, ‘field3’); to preload all those fields before outputting them. This enables you to load field1, field2 and field3 in 1 query rather than 3. On your first call to $page->field1 it won’t trigger a load from the database and instead will return the value that has already been preloaded. You can also call $page->preload(); without  any arguments, and it will preload ALL the supported fields for the page. 

In reality, ProcessWire is already pretty quick with loading fields, so you probably won’t benefit from preloading until the scale is quite large. While developing this, I was testing by iterating 500 pages and accessing 50 different fields from each. Without preload this took 12 seconds. With preload it took 6 seconds. So for this particular case, preloading cut the time in half. I’m not a query counter, as very often lots of simple DB queries are faster than a single big query, but I’ll mention that it also reduced the quantity of database queries by more than a factor of 10. For this large scale case, that meant more than 20000 queries down to well under 2000. 

Like with autojoin, there are some limitations with preloading. It supports primarily Fieldtypes that use the core loading mechanism. Some Fieldtypes (especially non-core) override this, and preload() skips over those fields. It also skips over most FieldtypeMulti (multi-row fields), but FieldtypePage is supported when used with Page fields that carry one value. Multi-value can be enabled for testing with an option you’ll find in the function $options, but like autojoin, is limited by MySQL’s group_concat limit. By default that limit is 1024, which supports 140-170 page reference values in a given page field. That's quite a lot, but I don't want to assume folks won't go over that, so it's disabled by default.

I’m guessing that most won’t need the preload() function, but a few might really benefit from it, especially at larger scale. So I think it’s a worthwhile addition to the core, and another method that answers a need that may arise as an installation grows, further supporting ProcessWire’s ability to scale up as needs do. Though consider it experimental and "work in progress" at the moment, as we’ll need to do more testing to make sure it is fully stable in a broader range of situations, and further improvements are likely. 

Special thanks to @tuomassalo at Avoine who came up with the idea for preload() and helped me to get started developing it. 

Last week I told you how Pete, Jan and I met up in Holland. I also wanted to mention, a couple weeks ago, right before I left for Amsterdam, Oliver from @update AG (update.ch) in Zürich, Switzerland sent me a DM saying that he was in my neighborhood, so we got together for coffee near my house. They’ve been using ProcessWire at Update AG almost as long as PW has been open source, so it was good to meet another long time ProcessWire user, and a nice coincidence that he was in the neighborhood. It’s always great to meet ProcessWire users in person and I hope to meet more of you in the future as well. Thank you for reading and have a great weekend! 

[wbmnfktr]

Is preload() already available in the DEV branch? I'm asking because the version is still 3.0.242.
What would be the best way to really test the performance boost here?
Need to try this in a recent project.

Maybe we should create some kind of meetups in the US and maybe even in the EU next year.
That would be fun.

[BitPoet]

13 hours ago, wbmnfktr said:

Maybe we should create some kind of meetups in the US and maybe even in the EU next year.
That would be fun.

Big thumbs up for that idea. If the timing is right, I might even make it to both, since I'm planning to head across the pond and hike the John Muir Trail next summer.

Love the preload method. I've got a shop solution built on PW with about 120k orders and 150k order positions by now, all as pages. Performance is becoming a bit of an issue in the backend with long range price change statistics and detailed Excel exports, and this seems like a perfect use case.

[TomPich]

On 11/8/2024 at 6:41 PM, wbmnfktr said:

Maybe we should create some kind of meetups in the US and maybe even in the EU next year.
That would be fun.

Yes, that would be great! Count me in (for Europe !) 😊