Webp support

[dotnetic]

@horst Thanks for the update and your effort on this. I assume, that when you are ready, a hook isn't required anymore, and we have a native API for getting the URL of the different formats?

[horst]

Yes! It's just for testing right now.

[jacmaes]

FYI: Today, January 29, Firefox 65 is out with WebP support. The only major browser left with no WebP support is Safari.

[StanLindsey]

This thread is really exciting. I'm looking forward to offering webp easily. =D Thanks @horst for your hard work.

[Ibrahim]

as my website is providing sport news articles, 

 

so i have made few changes over the templates for main article page 

 

i have added teh following : 

 

                      <?php $port = $pages->find("template=bns-article, limit=3, sort=-published"); ?>
                      <?php foreach ($port as $key2) {

	$imgOrigin = $key2->feature_img->url;
    $imgSplitted = $imgOrigin.'.webp';

$date1=date("H:i | d/m/Y", $key2->modified);
                  echo

and for thumb image i changed to the following.
                      <div class='boxImage lazy' style='background-image: url({$imgSplitted});'> </div>
                     <span class='news-time'>$date1</span>
                      <p class='n-title'>{$key2->title}</p>
                      <a href='{$key2->url}'
                          title='{$key2->title}' class='cards-link-overlay'>&nbsp;</a>
                  </div>";

 

instead of the main "Feature_img-" code, which is related in my case to the image upload box,

then i used Libwebp to convert all images to webp.

 

In my case, i got the following results : 

1- Thumbnail, all images converted to webp and the preview is working fine on my website.

2- When i upload webp image to a new article, its uploaded perfectly with no issue. 

 

 

i used different method as well, which is creating a new field for file upload, then i changed the main image source to this field name, where the result was accept webp file and previewed successfully 

[happywire]

Have webp support enabled as per this post. Thank you for that!

Now I have just installed TracyDebugger and if it is on I cannot upload a new image in the webp format.
No worries I can disable TracyDebugger when I upload new webp images to a page.

However it would be great if webp support can land in ProcessWire core somehow.
https://caniuse.com/#feat=webp looks great to me with the (obvious) exception being Safari.
But we now have Edge, Firefox, Chrome and Opera onboard, so that is great news.

[adrian]

@happywire - could you please look at the Network tab in the browser dev tools and monitor it for the file that is called via ajax that uploads the image. Could you check the response tab and see if there is an error appearing when Tracy is installed? Maybe also try Tracy in Strict Mode (button the Panel selector) and see if that helps / shows an error.

[Tom.]

Any news on WebP? There is a lot of convocation in the studio currently on image quality. The general thing is the built in resizing of images (Imagick) leaves images soft, however adding any sort of sharpening and it over sharpens around text leaving it with halos. 

Some people in the studio have been talking about changing CMS due to how ProcessWire manipulates images leaving them soft or over sharpened losing detail. It's a big discussion at the moment and generally we have found Statamic & CraftCMS much better at resizing images and keeping quality (Statamic uses Glide - https://glide.thephpleague.com/) 

But WebP will be massively important also. 

[dotnetic]

I also love to see some progression on WebP.

@Tom. Glide also utilizes GDLib or Imagick, so I don't know, if it will be an enhancement, as PW already does this too.

Do you have any examples of the reduced image quality? I did not notice quality loss, in terms of blur or oversharpening yet.

Are you also aware of the different settings for ImageSizer like sharpening or upscaling?

You can change these settings in your site/config.php

$config->imageSizerOptions = array(
	'upscaling' => true, // upscale if necessary to reach target size?
	'cropping' => true, // crop if necessary to reach target size?
	'autoRotation' => true, // automatically correct orientation?
	'interlace' => false, // use interlaced JPEGs by default? (recommended)
	'sharpening' => 'soft', // sharpening: none | soft | medium | strong
	'quality' => 90, // quality: 1-100 where higher is better but bigger
	'hidpiQuality' => 60, // Same as above quality setting, but specific to hidpi images
	'defaultGamma' => 2.0, // defaultGamma: 0.5 to 4.0 or -1 to disable gamma correction (default=2.0)
	);

 

[Tom.]

@jens.martsch

This is the example: https://www.dropbox.com/sh/ayqubd4b5o78i41/AAAm8eNmgBizOqfcG2YdT0Qta?dl=0
This is using quality 100, maxSize so no upscaling and sharpening none. 

[joshua]

What I've done so far, is a small implementation of webp-convert.
Currently without a ProcessWire specific module or something, but that's planned.

Just added a file called webp-on-demand.php in the ProcessWire root directory with following content:

<?php
// docs https://github.com/rosell-dk/webp-convert

require 'webp/webp-on-demand-1.inc';

use WebPConvert\WebPConvert;

$docRoot = rtrim($_SERVER["DOCUMENT_ROOT"], '/');
$requestUriNoQS = explode('?', $_SERVER['REQUEST_URI'])[0];
$source = $docRoot . urldecode($requestUriNoQS);

$destination = $source . '.webp';     // Store the converted images besides the original images (other options are available!)

$options = [

    // Tell where to find the webp-convert-and-serve library, which will
    // be dynamically loaded, if need be.
    //'reconvert' => true,
    'require-for-conversion' => 'webp/webp-on-demand-2.inc',

    'quality' => 'auto',
    'max-quality' => 75,
    'fail' => 'serve-original'
    // see https://github.com/rosell-dk/webp-on-demand/blob/master/docs/api.md for more info

];
WebPConvert::convertAndServe($source, $destination, $options);

Then in the folder webp the two on-demand files from here.

After that I added following to the .htaccess:

<IfModule mod_rewrite.c>
    RewriteEngine On
    # Redirect images to webp-on-demand.php (if browser supports webp)
    RewriteCond %{HTTP_ACCEPT} image/webp
    RewriteCond %{REQUEST_FILENAME} -f
    RewriteRule ^(.*)\.(jpe?g|png)$ webp-on-demand.php [NC,L]
</IfModule>

AddType image/webp .webp

 

And that's it. If the browser supports webp, all requests of .jpg and .png are redirectet to the webp-on-demand.php which desides if it needs to convert the image or serve the already converted one.
A small caveat: the standard installation of Plesk has no webp support included, so on webservers with Plesk you either need to wait for their implementation or compile it yourself.

 

 

 

[AndZyk]

Hello @Tom.,

do you use the core module IMagick Image Sizer and have you disabled sharpening in the module settings?

I usually disable it. Maybe that could be the reason. ?

Regards, Andreas

[horst]

On 4/24/2019 at 4:26 PM, Tom. said:

@jens.martsch

This is the example: https://www.dropbox.com/sh/ayqubd4b5o78i41/AAAm8eNmgBizOqfcG2YdT0Qta?dl=0
This is using quality 100, maxSize so no upscaling and sharpening none. 

@Tom. I have downloaded all four images and what you are saying, at least how I do understand it, seems not to be completly valid. What exactly was the source for what variation, as you said without resizing? (The original is larger than the others)

I found these images:

The original image is 4x larger (in pixel) than the compared variations. And it only has 157k, whereas the other have 300k.
The original image is from a source of 1920x1080 resized in Photoshop CC2017 to 1500x1074 px and is very strong compressed. That is NOT a useful choice for a master image! Masterimages should be saved in 100% quality, (or quality 12 in Photoshop), in every single item of a workflow chain. Once you saved with lesser quality, you damaged the image, regardless if you later on save it with higher quality.

When I save your original image in Photoshop with quality 12, I get a 427k image file, not 157k.

Another ProTip is: work with images in 16bit colordepth all the time, best in a larger color spectrum like AdobeRGB, not sRGB, use PSD or TIFF as your local master image, and only transform from this local master image to every desired target format according these steps, respecting the chronology:

 - 1) resize to target dimensions

 - 2) transform into target color space, (AdobeRGB into sRGB, or AdobeRGB into CMYK, etc)

 - 3) reduce the color depth from 16 bit to 8 bit, (the last step always!)

 - 4) save into your target file format, for ProcessWire (online) master images, with the max quality in JPEG.

From these (online) master images you can create very good results with the GD-Lib and with the IMagick-Lib. If you do not resize the original or only resize a very small value, you may set the sharpening property to "none". Also when using the GD-Lib, you may set the "defaultGamma" to -1. Experimenting with this two properties should be enough to render good results for all possible use cases. At least this are my experiences.

Can you tell me why the original is compressed so strong? And do you have an uncompressed version too? (uncompressed means: never ever compressed in its total live workflow chain!)

------------

There definitely is a difference when using the same sharpening values with GD-Lib and IMagick. This is due to the fact that IMagick internally already uses a sharpening algorithm when resizing images, and GD-Lib completely lacks about this feature.

Looking at PWs image resizing history, there first only was GD-Lib without sharpening, resulting in blurry images.
Than we introduced a sharpening algorithm with four available values: none | soft | medium | strong, (defaults to soft)

A long time later, we introduced the ImageMagick-Lib with all the same params available as for the GD-Lib. 
At this time my thinking was, when switching from GD to IMagick, one may switch from medium to soft, or from soft to none in the site/config.php file, as it is a global decision what image library to use. But a review / rework of the used algorithm values in the core libs is possible, so that we maybe will have nearly no differences between IMagick none and soft, but keep the currently used differences within the GD-Lib.

 

[horst]

5 hours ago, joshua said:

And that's it. If the browser supports webp, all requests of .jpg and .png are redirectet to the webp-on-demand.php which desides if it needs to convert the image or serve the already converted one.

Hi @joshua, a nice and global solution.

But want to say that invoking the PHP engine for "every" image on a website may result in to much serverload. This in mind, it is a nice solution for sites with a smaller amount of images. ?

[joshua]

Hi @horst,

that's right, but the script looks, if there is already a webp-version of the image and if deliver this directly. So there is no big processing during normal frontend activities through website users, if the conversion was already done. Though you are definetely right, it's an extra PHP call (but cacheable)

[horst]

7 minutes ago, joshua said:

Though you are definetely right, it's an extra PHP call (but cacheable)

Yes, once the webp variation is created and cached, it only will be served, but the PHP engine is invoked for every image call. This makes a difference compared to simply serve a static file directly by apache on sites with lots of images, like archives. Thats why I said it is a nice solution for sites with smaller amounts of images.

For example: I maintain sites with 60k+ images, that presents 100 thumbs per page and have infinite scrolling, that may result fastly in 100 to 1000 thumbs on a single page view. There it makes a difference, if apache serves 1000 thumbs from static files, or if the webserver has to invoke 1000 times the PHP engine for serving static files. And multiply this with a decend amount of simultaneous visitors of the site, I bet you too definetly want to avoid invoking the PHP engine for serving static image files.

[Tom.]

5 hours ago, horst said:

@Tom. I have downloaded all four images and what you are saying, at least how I do understand it, seems not to be completly valid. What exactly was the source for what variation, as you said without resizing? (The original is larger than the others)

I found these images:

The original image is 4x larger (in pixel) than the compared variations. And it only has 157k, whereas the other have 300k.
The original image is from a source of 1920x1080 resized in Photoshop CC2017 to 1500x1074 px and is very strong compressed. That is NOT a useful choice for a master image! Masterimages should be saved in 100% quality, (or quality 12 in Photoshop), in every single item of a workflow chain. Once you saved with lesser quality, you damaged the image, regardless if you later on save it with better quality.

When I save your original image in Photoshop with quality 12, I get a 427k image file, not 157k.

Another ProTip is: work with images in 16bit colordepth all the time, best in a larger color spectrum like AdobeRGB, not sRGB, use PSD or TIFF as your local master image, and only transform from this local master image to every desired target format according these steps, respecting the chronology:

 - 1) resize to target dimensions

 - 2) transform into target color space, (AdobeRGB into sRGB, or AdobeRGB into CMYK, etc)

 - 3) reduce the color depth from 16 bit to 8 bit, (the last step always!)

 - 4) save into your target fileformat, for ProcessWire (online) master images, with the max quality in JPEG.

From these (online) master images you can create very good results with the GD-Lib and with the IMagick-Lib. If you do not resize the original or only resize a very small value, you may set the sharpening property to "none". Also when using the GD-Lib, you may set the "defaultGamma" to -1. Experimenting with this two properties should be enough to render good results for all possible usecases. At least this are my experiances.

Can you tell me why the original is compressed so strong? And do you have an uncompressed version too? (uncompressed means: never ever compressed in its total live workflow chain!)

------------

There definetly is a difference when using the same sharpening values with GD-Lib and IMagick. This is due to the fact that IMagick internally already uses a sharpening algorythm when resizing images, and GD-Lib completly lacks about this feature.

Looking at PWs image resizing history, there first only was GD-Lib without sharpening, resulting in blurry images.
Than we introduced a sharpening algorythm with four available values: none | soft | medium | strong, (defaults to soft)

A long time later, we introduced the ImageMagick-Lib with all the same params available as for the GD-Lib. 
At this time my thinking was, when switching from GD to IMagick, one may switch from medium to soft, or from soft to none in the site/config.php file, as it is a global decision what image library to use. But a review / rework of the used algorythmvalues in the core libs is possible, so that we maybe will have nearly no differences between IMagick none and soft, but keep the currently used differences within the GD-Lib.

 

Thank you very much for the explanation, this is really useful information. I provided the images provided to me by the designers and forward their complains on image quality. However, this has always been a difficult argument for me as I only look at it technically. I can now pass this information on to them and make sure they are outputting images correctly.

Thanks again, I've worked on a small webP converter in the meantime, until we see something implemented into the core.

[horst]

I have sent a pull request on github for webP support in the core: https://github.com/processwire/processwire/pull/141

Explanation of its usage:

I implemented two new options for the imagesizer that can be set globally in the site/config.php or can be passed within an individual options array to any size-method:

$options = [
	"webpAdd" => true,   // boolean, if true, an additional webp variation will be created
	"webpQuality" => 80  // integer range 0 - 100, 100 is best
];

This creates an additional webp variation file besides the regular JPEG, PNG or GIF variations. 

In the template files you have two new image properties to work with the webp variation: 

$image->hasWebp;    // boolean, true indicates that a webp variation is available for that image variation
?>
<img src="<?= $image->srcWebp ?>" alt="" />

So we can create conditional markup like in this example: 

$image = $page->images->first()->size(300, 300);

?>
<picture>
    <?php if($image->hasWebp) { ?>
    <source srcset="<?=$image->srcWebp?>" type="image/webp" />
    <?php } ?>
    <source srcset="<?=$image->src?>" type="image/jpeg" />
    <img src="<?=$image->url?>" alt="Alt Text!" />
</picture>

The pull request and my test branch are based on this weeks pw dev branch. So if someone want to test it, please grab a copy, play with it and report back here.

I don't know what timeline @ryan has and what parts need to be modified / rearanged, etc. Maybe the property names or the option names will be changed, but I think the main functionality will be close to that in the current test branch.

-----

And here are a filesize comparision with different quality settings and with both engines, GD-Lib and IMagick:

EDIT:

An approach without conditional markup for webp comes from @arjen see lazyload.

...

Edited April 25, 2019 by horst

[dotnetic]

Thanks @horst. Hopefully @ryan will include this quickly into the core.

[Tom.]

@horst

Thank you so much, I'm playing with this now - I'll let you know if I come across anything. Appreciate the hard work.

Edit: Looking good!

 

Edit: 

One thing I've noticed is let's say you set the quality of the jpeg to 50 on resize, WebP will use the resized jpeg at 50 quality. Does this mean that using also webpQuality 50 would compress it twice? @horst

[horst]

3 hours ago, Tom. said:

One thing I've noticed is let's say you set the quality of the jpeg to 50 on resize, WebP will use the resized jpeg at 50 quality. Does this mean that using also webpQuality 50 would compress it twice?

@Tom. Many thanks for your feedback!

And, uhm, - GD does not use the compressed jpeg for webp creation, but currently IMagick does. But this is not intended and I locally have a version that uses the uncompressed src variation for webp creation. I will finalize this and commit an update to the test branch later this evening.

So the intended usage is that you can define independend quality settings for jpeg and webp, and both final variations will be created from the uncompressed variation bitimage.

["quality" => 80; "webpQuality" => 90]

 

If you simply want to test out GD-Lib when IMagick is installed with higher priority, you can pass the param "forceEngine" with the options array to the size method:

        $options = [
            'forceEngine' => 'ImageSizerEngineGD',  // ImageSizerEngineGD | ImageSizerEngineIMagick
            'forceNew' => true,
            'webpAdd' => true,
            'webpQuality' => 90,
            'quality' => 80,
            'sharpening' => 'none'
        ];

 

[Noboru]

@horst, it's perfect for me. I already had WebP since your code from late 2018 and use my own ImageSizer, which relies on LibVips. The Vips engine is very fast - about 5 times faster than ImageMagick. I hope WebP will make it into the core soon - many thanks for your hard work!

[horst]

I have played with a .htaccess only solution to serve WEBP instead of JPEGs or PNGs.

This way you can
 - leave your existing template markup untouched,
and the apache_rewrite will
 - detect if the browser supports WEBP
 - if a WEBP copy is available for a JPEG or PNG image.

Locally it's working out nicely! ?

This markup automatically serves a WEBP copy to all browsers that support it:

$options = ['webpAdd'=>true, 'webpQuality'=>90];
$image = $page->images->first()->size(300, 300, $options); // this will create a 300x300 Thumb for JPEGs and PNGs with an additional WEBP copy

?>

<img src="<?=$image->src?>" alt="<?=$image->alt?>" />  
<!-- this will serve the WEBP image to all Browsers that supports it, and the JPEG/PNG to other browsers -->

AddType image/webp .webp

<IfModule mod_rewrite.c>

    RewriteEngine On
    AddDefaultCharset UTF-8

    ### Redirect regular PW JPEGs and PNGs to their WEBP image copies,
    ### if available and if the Browser supports WEBP:

        ## Does Browser accept WEBP images?
        RewriteCond %{HTTP_ACCEPT} image/webp

        ## Is it an existing file?
        RewriteCond %{REQUEST_FILENAME} -f

        ## Does a WEBP copy exist for it?
        RewriteCond %{DOCUMENT_ROOT}/$1$2$3/$4.webp -f

        ## With an added GET var or GET value of skiprewrite we do send the original JPEG or PNG
        RewriteCond expr "! %{QUERY_STRING} -strmatch '*skiprewrite*'"
        ## With an added GET var or GET value from the PW Page Editor, we do send the original JPEG or PNG
        RewriteCond expr "! %{QUERY_STRING} -strmatch 'nc=*'"

    ## Is it a regular PW JPEG or PNG image, stored under site/assets/files?
    RewriteRule ^(.*?)(site/assets/files/)([0-9]+)/(.*)\.(jpe?g|png)(.*)$  /$1$2$3/$4.webp [L]

</IfModule>

Adding a GET var or GET value with the string "skiprewrite" added to an image bypasses serving the WEBP copy in favour for the original variation. (Maybe useful for debug purposes!)

Edited April 27, 2019 by horst
again modified the .htaccess a bit

[horst]

On 4/25/2019 at 2:38 PM, Tom. said:

One thing I've noticed is let's say you set the quality of the jpeg to 50 on resize, WebP will use the resized jpeg at 50 quality. Does this mean that using also webpQuality 50 would compress it twice?

Hi @Tom.

the new code is online in my test branch. And following is a test with also the .htaccess solution installed.

My template code:

<?php
$image1 = $page->images->first->size(100, 100, $options);
?>

<img src="<?=$image1->url?>" alt="<?=$image1->alt?>" /> <img src="<?=$image1->url?>?skiprewrite" alt="<?=$image1->alt?>" />

<?php
$image2 = $page->images->first->size(100, 100, $options);
?>

<img src="<?=$image2->url?>" alt="<?=$image2->alt?>" /> <img src="<?=$image2->url?>?skiprewrite" alt="<?=$image2->alt?>" />

 

The quality settings for the regular variation is set to 10, it is created first in the image engines. The webpQuality was set to 100, and it is created after the regular variation. The following screen first shows the webp copy and due to the "?skiprewrite" GET varname, it shows the (distorted) jpegs at second place. Once with GD and once with IMagick:

 

I really love the .htaccess only solution! With this, you simply can update any existing site by upgrade to the new PW core that include the webp support and running a maintenance script over all image variations to create the webp copies. After that you are done! ?

 

[horst]

21 hours ago, Noboru said:

and use my own ImageSizer, which relies on LibVips.

Hi @Noboru, is your own ImageSizer written as a pw image rendering engine, or do you use the hooks like in the times before the pw image rendering engines were implemented?