Delayed Image Variations

[Robin S]

Delayed Image Variations

Delays the creation of image variations until their individual URLs are loaded.

Image variations being generated one-by-one:

Background

Normally when you create new image variations in a template file using any of the ProcessWire image resizing methods, all the new variations that are needed on a page will be created from the original Pageimage the next time the page is loaded in a browser. If there are a lot of images on the page then this could take a while, and in some cases the page rendering might exceed your max_execution_time setting for PHP.

So you might like to have image variations be generated individually when they are requested rather than all at once. That way the page will render more quickly and the risk of a timeout is all but eliminated. But there can be problems with some implementations of this approach, such as with the (in)famous TimThumb script:

• It's not ideal to have PHP be involved in serving every image as this is needlessly inefficient compared to serving static assets.
• It's not good to allow arbitrary image sizes to be generated by varying URL parameters because you might want to restrict the maximum resolution an image is available at (e.g. for copyrighted images).
• If images are generated from URL parameters a malicious user could potentially generate thousands of images of slightly different dimensions and fill up all your disk space.

The Delayed Image Variations module avoids these problems - it creates variations when their URLs are loaded but only allows the specific dimensions you have defined in your code. It does this by saving the settings (width, height and ImageSizer options) of every new Pageimage::size() call to a queue. The module intercepts 404s and if the request is to an image variation that doesn't exist yet but is in the queue it generates the variation and returns the image data. This only happens the first time the image is requested - after that the image exists on disk and gets loaded statically without PHP.

Usage

In most cases usage is as simple as installing the module, and you don't need to change anything in your existing code.

However, there might be some cases where you don't want the creation of a particular image variation to be delayed. For example, if you created a variation in your code and then immediately afterwards you wanted to get information about the variation such as dimensions or filesize.

$resized = $page->image->width(600);
echo $resized->height;
echo $resized->filesize;

This wouldn't work because the actual creation of the resized image hasn't happened yet and so that information won't be available. So in these cases you can set a noDelay option to true in your ImageSizer options and Delayed Image Variations will skip over that particular resizing operation.

 

$resized = $page->image->width(600, ['noDelay' => true]);
echo $resized->height;
echo $resized->filesize;

For advanced cases there is also a hookable method that you can return false for if you don't want a delayed variation for any particular resizing operation. Example:

 

$wire->addHookAfter('DelayedImageVariations::allowDelayedVariation', function(HookEvent $event) {
	/** @var Pageimage $pageimage */
	$pageimage = $event->arguments(0); // The Pageimage to be resized
	$width = $event->arguments(1); // The width supplied to Pageimage::size()
	$height = $event->arguments(2); // The height supplied to Pageimage::size()
	$options = $event->arguments(3); // The options supplied to Pageimage::size()

	// Don't delay variations if the Pageimage belongs to a page with the product template
	if($pageimage->page->template == 'product') $event->return = false;
});

404 handling

For Delayed Image Variations to work your .htaccess file needs to be configured so that ProcessWire handles 404s. This is the default configuration so for most sites no change will be needed.

# -----------------------------------------------------------------------------------------------
# 2. ErrorDocument settings: Have ProcessWire handle 404s 
#
# For options and optimizations (O) see: 
# https://processwire.com/blog/posts/optimizing-404s-in-processwire/
# -----------------------------------------------------------------------------------------------

ErrorDocument 404 /index.php

ProCache

If you are using ProCache then make sure it is not configured to cache the 404 page or else PHP will not execute on 404s and queued image variations will not be generated.

Generate queued variations

Before launching a new website you might want to pre-generate all needed image variations, so no visitor will have to experience a delay while a variation is generated. To queue up the image variations needed for your site you will need to visit each page of the website one way or another. You could do this manually for a small site but for larger sites you'll probably want to use a site crawler tool such as Xenu's Link Sleuth. This may generate some image variations but it's likely that some other variations (e.g. within srcset attributes) will not be requested and so will remain queued.

To generate all currently queued variations there is a button in the module config:

This will search the /site/assets/files/ directory for queue files and render the variations.

 

https://github.com/Toutouwai/DelayedImageVariations
https://processwire.com/modules/delayed-image-variations/

[monollonom]

Okay I have to try it out but this is absolutely fantastic and something I wanted to tackle myself for a while now, working on websites with a lot of images.

Thanks for sharing this!

[Jozsef]

Thanks a million, I know it will be a staple for me from now on.

[Jozsef]

I'm not sure but is that a mistake in the readme? The image, width, height and options all equal to $event->arguments(0) in the hook example.

[Robin S]

4 minutes ago, Jozsef said:

I'm not sure but is that a mistake in the readme? The image, width, height and options all equal to $event->arguments(0) in the hook example.

Thanks, fixed now.

[bernhard]

This is brilliant @Robin S - very clever solution ?? Should be merged into the core imho once it is more tested!

[bernhard]

Hey @Robin S this is working great so far ? I have one issue when using RockMigration's filesOnDemand feature.

That feature hooks into Pagefile::url and Pagefile::filename and if the image does not exist it tries to grab it from the remote server. This is extremely handy when using "rockshell db:pull", because then you have a copy of the remote database on your local dev environment which could have several references to images in the database that do not exist on the filesystem. RockMigrations then downloads those images from the remote server as soon as you visit a page with images that do not exist.

This feature can be enabled at runtime by setting $config->filesOnDemand = false; Do you think you can add this to your module to make it work with RockMigrations? Or do you see another way to make it work out of the box? Maybe I can check if the item is in the queue before trying to download it from the remote.

See the code here https://github.com/baumrock/RockMigrations/blob/9d9609904647e7b2cfc14a81b2da411edfdca814/RockMigrations.module.php#L2536-L2583

Thx for your thoughts!

[Robin S]

I made some updates in v0.1.1 so that unprocessed queue files get deleted if the original Pageimage is deleted.

21 hours ago, bernhard said:

Maybe I can check if the item is in the queue before trying to download it from the remote.

That sounds like a simple solution. In v0.1.1 the queue file is just the variation URL with ".txt" at the end. So for a variation that will have a URL "/site/assets/files/1234/myimage.500x500.jpg" once it is created the queue file would be at "/site/assets/files/1234/myimage.500x500.jpg.txt". So in your loadFilesOnDemand() method you could check if DelayedImageVariations is installed and if so check for the existence of the queue file in addition to the variation file.

@bernhard, in v0.1.2 I've changed the queue file extension to ".queue" because it will allow me to identify the queue files more efficiently for another feature I have in mind to release soon. So the section in strikethrough above will now be:
In v0.1.2 the queue file is just the variation URL with ".queue" at the end. So for a variation that will have a URL "/site/assets/files/1234/myimage.500x500.jpg" once it is created the queue file would be at "/site/assets/files/1234/myimage.500x500.jpg.queue".

[Stefanowitsch]

@Robin S

THIS IS A BLAST!

I just testet it on a image-heavy site and it works perfectly. Thanks a lot!!!

This definitely belongs  to the core of PW in a newer version.

[cb2004]

So so good. Can't wait to put this on a photographers site tomorrow. 

[Robin S]

Version 0.1.3 is released. This adds a button to generate all currently queued variations:

Quote

Generate queued variations

Before launching a new website you might want to pre-generate all needed image variations, so no visitor will have to experience a delay while a variation is generated. To queue up the image variations needed for your site you will need to visit each page of the website one way or another. You could do this manually for a small site but for larger sites you'll probably want to use a site crawler tool such as Xenu's Link Sleuth. This may generate some image variations but it's likely that some other variations (e.g. within srcset attributes) will not be requested and so will remain queued.

To generate all currently queued variations there is a button in the module config:

This will search the /site/assets/files/ directory for queue files and render the variations.

The readme is also updated to add some important notes about 404 configuration in .htaccess and ProCache configuration.

[d'Hinnisdaël]

Amazing, thanks for creating this! I agree this should be part of the core once it's been in use for a while.

[Fuzen]

Doesn’t PW’s image size create and save the various image sizes on the first load of the page? If so, then on smaller sites this module wouldn’t be of big benefit, am I correct?

[Robin S]

On 11/18/2023 at 2:13 PM, Fuzen said:

Doesn’t PW’s image size create and save the various image sizes on the first load of the page?

Yes, that's mentioned in the readme:

On 11/10/2023 at 9:48 PM, Robin S said:

Normally when you create new image variations in a template file using any of the ProcessWire image resizing methods, all the new variations that are needed on a page will be created from the original Pageimage the next time the page is loaded in a browser.

 

On 11/18/2023 at 2:13 PM, Fuzen said:

If so, then on smaller sites this module wouldn’t be of big benefit, am I correct?

I'd say it's not the size of the site that matters but the number of image variations on any particular page. But in any case, if the way variations are created by default in PW is not presenting a problem then you won't need the module.

[bernhard]

A note for anybody using RockMigrations deployments: You have to add one line to your /site/deploy.php file:

<?php

namespace RockMigrations;

require_once __DIR__ . "/modules/RockMigrations/classes/Deployment.php";
$deploy = new Deployment($argv);

// add this line
$deploy->share("/site/assets/div-queue"); // DelayedImageVariations

$deploy->run();

Otherwise the queue-folder does not exist after the next deployment and the module throws errors, because the module creates this folder only on installation.

[Robin S]

@bernhard, the /site/assets/div-queue/ directory was only used in v0.1.0. From v0.1.1 onwards the queue files are distributed in the same location as the variation files, e.g. /site/assets/files/1234/, and so the /site/assets/div-queue/ directory isn't needed.

[bernhard]

Thx! Looks like I should update ? And great to hear that as this plays well with RockMigration's deployment ?

[Stefanowitsch]

@Robin S I have a question:

On the latest sites that are using this module all of a sudden the image URLs get crawled by google and they appear in the search console (they are not indexed, though).

This is a bit of a strange behaviour, I've never seen this on my older websites. Does this have anything to do with the way this modules handles the delayed image generation?

Quote

The module intercepts 404s and if the request is to an image variation that doesn't exist yet but is in the queue it generates the variation and returns the image data. This only happens the first time the image is requested - after that the image exists on disk and gets loaded statically without PHP.

 

[Robin S]

11 hours ago, Stefanowitsch said:

On the latest sites that are using this module all of a sudden the image URLs get crawled by google and they appear in the search console (they are not indexed, though).

I can't think of any reason or way that this module could be related to what Google crawls, so I think the timing must be a coincidence and the cause is something else.

[dynweb]

Hi Robin,

With the module installed, when I want to edit an image (e.g. crop) in admin, I get the following exception on clicking the "Apply" button:

The exception also occurs if I have previously generated all delayed images with the option "Generate queued variations". The error does NOT occur after uninstalling the module.

Any ideas? Thanks ?

[Robin S]

12 hours ago, dynweb said:

With the module installed, when I want to edit an image (e.g. crop) in admin, I get the following exception on clicking the "Apply" button

Thanks for the report, should be fixed in v0.1.4.

[nurkka]

Thanks @Robin S for this great module – and all your other great modules, too! I am using really a lot of them, and the number is still growing 🙂

DelayedImageVariations works perfectly with JPEG images. But unfortunately, it does not work in my case, because I am using WEBP images, like so:

$image_url = $image_object->size(1200,400)->webp()->url;

So I get real 404 pages for the generated WEBP-URLs, which result in empty image frames on my website.

Is this the expected behaviour, i.e. is WEBP not supported by DelayedImageVariations? Or am I doing something wrong?

Additional Info: I am currently not using the .htaccess WEBP to JPEG fallback stuff, which is described here https://processwire.com/blog/posts/webp-images-and-more/#strategy-2-delivering-webp-and-automatically-falling-back-to-jpg-png , because I am using <picture>-tags with srcset everywhere. And also the webpAdd option in $config->imageSizerOptions is set to `false`, which I believe is the default. My site also doesn't use any caching at all.

[ Also I just found out, that the "Render Queued Variations" Feature somehow generates the missing WEBP images, too – even if they are not displayed on the "Render Queued Variations" output page, where only the JPEGs are displayed. Although it once takes a long time to reload the frontend page, after using "Render Queued Variations". So it could also be that the WEBPs are generated only on the frontend. But they won't be generated without hitting "Render Queued Variations" first. ]

[Robin S]

@nurkka, when I test here DelayedImageVariations is sort of working with webp(), although I'm surprised it works at all because I thought it would be a case like the one mentioned in the readme:

On 11/10/2023 at 9:48 PM, Robin S said:

$resized = $page->image->width(600);
echo $resized->height;
echo $resized->filesize;

This wouldn't work because the actual creation of the resized image hasn't happened yet and so that information won't be available.

I thought the webp() method would need an actual resized variation to be available at the time it is called.

But I'm testing like this...

foreach($page->images as $image) {
	$image_url = $image->size(250,250)->webp()->url;
	bd($image_url, "image_url");
	echo "<img src='$image_url' alt=''>";
}

...and what I experience is that on the first page load the webp()->url results in a JPG file...

...and on subsequent page loads it results in a WebP file. 

 I don't get any 404s or broken images, and the only effect is slightly larger file sizes on the first page load, which isn't that big of a deal.

But if you experience something different then I don't have a solution within DelayedImageVariations, sorry. My suggestion then would be to pursue a different strategy to pre-generate your image variations rather than use DelayedImageVariations. You could generate the variations on page save. See this post for how you could do it without slowing down page saves in the admin: