Crowdfunded Tinypng Integration Module

[OllieMackJames]

EDIT after couple of days on the forum:

Thanks all for your thoughts, we only raised 200$, and no one offered to code it

since I am not a coder, only a very happy user of the great processwire cms

I guess this will not happen - at least not now.

Great discussion on the technical side of things, thanks for that!

CROWDFUNDED SO far:

OllieMackJames 100$

Peter Knight 100$

=================shout out below and I'll add your amount here============================

I would love to see a tinypng/tinyjpg integration module for processwire. Main focus is to get solid and good image compression, nothing else.

There is already a php api: https://tinypng.com/developers/reference/php

Who else is interested in this?

Who would like to build it?

What I would like it to do is the following:

- have a backend where image compression settings can be changed/set

- image compression upon upload

- image compression of already uploaded files

- procache integration

- no interference with other modules like thumbnails or croppableimage (they should work with the optimized images)

Please reply if you would like this module to happen, and how much you can contribute to the development costs, once ready the module will be given to the community

then also reply your wishes.

If you would like to build the module, please give a quote so we - the crowd - know how much to fund!

[horst]

@OllieMackJames: interesting thing.

There are some thoughts that comes to my mind directly in regard for an understanding how imageprocessing works in PW:

• there is already a service integrated for minifying images (jpeg and png): ProcessImageMinimize.pw
   
• it is not possible to further manipulate minimized images, as I assume you meant with other modules (thumbnails etc) should work with the minimized images. This will lead in very very bad quality and sometimes corrupt looking images.
  It is mandatory to create every variation from the original, un-minified image, or at least from an image that was not compressed and has 100% quality, what is not the same like the original image.
   
• So, it is mandatory to send every image variation as last step of the processing chain to the minimize service (or a local program)
   
• Compression of the original image on upload is not possible if you want to create any variations from it. (see above!)

Hope this helps a bit.

PS: There was also a module that handles local optimizing of jpegs (OptimJpeg). Maybe we should update this to also support png, by asking the author or by forking it?

[matjazp]

@OllieMackJames: I would love to have that module. I can help with testing/debugging as I'm not that good at programming.

@Horst: I know ProcessImageMinimize.pw, but it is not free (although they have free plan). ProcessImageMinimize is not supported on PW3 (I asked Marvin Scharle at Conclurer). I tested it and couldn't make it work (running on Windows with IIS - maybe this is the culprit). And probably there will be no more support for it.

Module JpegOptimImage is using exec, some sites do not allow that. Most of my users doesn't know anything about images so they upload just what they have, even if those are big images (like 5-10 Mb) and large sizes. I just go after them, resize if needed (yes, I have max/min width set in ImageField) and then optimize with jpegmini. 

[horst]

 @Horst: I know ProcessImageMinimize.pw, but it is not free (although they have free plan). ProcessImageMinimize is not supported on PW3 (I asked Marvin Scharle at Conclurer). I tested it and couldn't make it work (running on Windows with IIS - maybe this is the culprit). And probably there will be no more support for it.

Module JpegOptimImage is using exec, some sites do not allow that. Most of my users doesn't know anything about images so they upload just what they have, even if those are big images (like 5-10 Mb) and large sizes. I just go after them, resize if needed (yes, I have max/min width set in ImageField) and then optimize with jpegmini. 

@matjazp: Thanks for clarifying. I didn't know that there is no further maintainance of the minimize service.

I also know that we would need CLI processing via PHPs exec(), and that this isn't supported on (some) shared hosts. But also the bullet proofed method would be to process variations and minifying locally. A less bullet proofed method would be to use "a sort of" remote service via http on the own host, if possible, because you also have full control and the http protocoll overhead is small and results in appropriate fast speed. The last one in the hirarchy is to use a foreign remote service, as you have no control and a lot of overhead.

I'm interested in how many shared hosts have the ability to choose php fastcgi handler for subdirectories (via htaccess), where you than have support for exec(). I'm on this sort of shared servers myself and use PHP as apache_module with disabled exec() as the (main) PHP handler for the whole site. But I can define fastcgi handler(s) for A) a specific subdirectory or B) for a specific defined file extension. This way I'm able to run my PW site within a fast apache_module PHP handler and pass the final step of optimization via http to a cgihandler script. This is really fast, as I do not have to sent and retrieve files, I also do not need to resolve a DNS for the connection, as it is on the same (local) host. If there is the possibilty for others too, I'm happy to share my solution.

[teppo]

Just wanted to say that this sounds very interesting. Been meaning to build a module like this, though was thinking of limiting it to PNGs since that seems to be the only case where optimization doesn't necessarily mean sacrificing quality.

I'm also aware of the minimize.pw service, but a) it's not free (as in beer or as in freedom), b) it doesn't seem to be well maintained and c) I don't want to rely on external services unless I really trust them (especially when it means sending client data over to an external server).

As a reply to Horst, kind of, I'd like to point out that whether or not the host allows exec(), if there's a similar solution that doesn't require it, that's in my opinion always the better route. Even when you're taking every step to make sure that it's safe, exec() is still potentially dangerous. Any and all mistakes have the potential to compromise your entire server (or, at the very least, your personal account).

There's a reason why exec() and other code execution methods tend to trigger warnings from security scanners. That being said, I won't deny that executing external apps via PHP can solve some situations where nothing else helps. I've had to use it many times over myself. 

Edit: just took a closer look at Tinify. Somehow I managed to miss earlier the point that it is also an API, i.e. requires sending the data to their servers for processing. Can't say that I would be exactly happy with that, but will give this a while; perhaps it's a compromise worth making. In the meantime I'll be looking into some alternative approaches.

Edited February 2, 2016 by teppo

[kongondo]

Edit: just took a closer look at Tinify. Somehow I managed to miss earlier the point that it is also an API, i.e. requires sending the data to their servers for processing. Can't say that I would be exactly happy with that, but will give this a while; perhaps it's a compromise worth making. In the meantime I'll be looking into some alternative approaches.

Hmm, not so sure about that...Have a read here:

Hook up your server to TinyJPG and TinyPNG and compress all your images on the fly. The API uses the compression engine that powers the web services so you can expect exactly the same results. Enter your name and email address below to retrieve your key and get started.

Am  I missing something? I read that as 'there is a web service' (see pricing at the bottom of that page) and an API that you can use separately? (hosted on GitHub)

Edit: You are right @teppo, going by this WP plugin....

Edit 2: Practical question. Once the 'module' is completed by 'x'...who would maintain it? 'x' or the community? Is there a situation where 'x' would need to be 'compensated' for continuing to maintain a module on behalf of the community? Just wondering...

Edited February 2, 2016 by kongondo
edit 2

[kongondo]

Just to qualify my question above, I'm just trying to think through the whole idea of 'crowd-funded' modules, since, IIRC, it is uncharted territory for us (although quite interesting). Just wondering if it is something we need to think through a bit more.?..or am just being too fussy.

OK, back to my hiding place...

[OllieMackJames]

@horst

Thanks for chiming in. I never got minimize.pw to work tried contacting them number of times, never got any reply, so I figure that is not the route.

Thanks for the other input - OptimJpeg could be an option

@matjaZP great to hear you are interested, how interested in terms of crowdfunding to pay a coder?

@horst

I also know that we would need CLI processing via PHPs exec(), and that this isn't supported on (some) shared hosts. But also the bullet proofed method would be to process variations and minifying locally. A less bullet proofed method would be to use "a sort of" remote service via http on the own host, if possible, because you also have full control and the http protocoll overhead is small and results in appropriate fast speed. The last one in the hirarchy is to use a foreign remote service, as you have no control and a lot of overhead.

I'm interested in how many shared hosts have the ability to choose php fastcgi handler for subdirectories (via htaccess), where you than have support for exec(). I'm on this sort of shared servers myself and use PHP as apache_module with disabled exec() as the (main) PHP handler for the whole site. But I can define fastcgi handler(s) for A) a specific subdirectory or B) for a specific defined file extension. This way I'm able to run my PW site within a fast apache_module PHP handler and pass the final step of optimization via http to a cgihandler script. This is really fast, as I do not have to sent and retrieve files, I also do not need to resolve a DNS for the connection, as it is on the same (local) host. If there is the possibilty for others too, I'm happy to share my solution.

I would love to test this - I am on my own vps so can do what I like here

@teppo so you could be game to code it then? My preference would be to have a local solution as well, but if needed paid is fine as well, as long as the results are good

@kongondo, great question on the practical side of things, IMHO if x(the coder) could do the support, that would be great, if not then perhaps the community, but paying x for the updates would be fine as well for me.

Thanks again for all who chimed in!

[OllieMackJames]

Great to ask others to crowdfund, so let me start myself:

I am in for 100$ - if you think this is a great thing and want to see it happen, shout out and I'll add your amount to the top post!

[Peter Knight]

Great to ask others to crowdfund, so let me start myself:

I am in for 100$ - if you think this is a great thing and want to see it happen, shout out and I'll add your amount to the top post!

I'll contribute too. ($100). Would love to see an integrated image optimisation module if there's agreement on the fundamentals.

I know there are some upcoming improvements with the image UI on the horizon. I wonder are there plans for this type of service as a native or paid module.

I also think that both support and maintenance need consideration.

Some good approaches here

http://www.mobiloud.com/blog/2015/02/10-best-image-optimization-plugins-speed-wordpress-site/

[teppo]

Just to qualify my question above, I'm just trying to think through the whole idea of 'crowd-funded' modules, since, IIRC, it is uncharted territory for us (although quite interesting). Just wondering if it is something we need to think through a bit more.?..or am just being too fussy.

From my point of view it doesn't matter that much, as long as the work is good and it is released for the general public under a proper open source license. We already have examples of someone else taking over a module that another person started, etc.

To perhaps answer your question better, I tend to see the crowdfunding part mainly as an addition to current workflow where interested parties build and release modules for completely free. In this case you get paid for that initial work.

If the module needs maintaining in the "fix bugs and make sure it works with new versions" kind of way, these should imho be provided for free as much as possible (or by accepting contributions from other interested parties). If the original developer can't handle this and the module has valid use cases, I'm sure we can find someone else who does have the time to do it.

In the case of "nice module, but I think it needs big feature x", there's always the option of arranging another round of crowdfunding

@teppo so you could be game to code it then? My preference would be to have a local solution as well, but if needed paid is fine as well, as long as the results are good

It's one option and I'm definitely interested, but like I said, I don't have much time on my hands right now; you might find it faster and easier to catch another developer for this

[matjazp]

@matjaZP great to hear you are interested, how interested in terms of crowdfunding to pay a coder?

I've done 3 sites with PW, all are educational, nonprofit and I don't get payed for that so crowdfunding would have to go from my personal/family budget...

[horst]

Edit: just took a closer look at Tinify. Somehow I managed to miss earlier the point that it is also an API, i.e. requires sending the data to their servers for processing. Can't say that I would be exactly happy with that, but will give this a while; perhaps it's a compromise worth making. In the meantime I'll be looking into some alternative approaches.

Yes, it also is an API, and it is not free, only the first 500 image (variations!) per month.

So, for me that's exactly the point why I hasitated to say I'm interested in working on this. As it is no problem for me to name the exact and best points where and how to hook into PW's image procesing chain for this, I'm not comfortable with the part that needs to be build that, for example, observes the amount of total submissions in a given paid or free plan per month, or should be there to detect if something went wrong, (e.g. a script goes wild) you can reach a 500 limit in under one hour or shorter.  (500.000 variations = $1065.50 and max 5.000.000 variations = $10065.50)

For me, the integration in the image processing chain looks easy, but I assume that this will take only 20%-30% of the whole module. 70%-80% will be taken by the required "overhead".

@teppo: Regarding exec()

AFAIK for imageprocessing without exec(), there is the bundled GD-lib and the Imagick PHP extension. And than a lot of other solutions that all rely on CLI processing. So, it doesn't to be not uncommon to use it for image processing . It needs to be integrated into the right server configuration, I believe. On the servers I've spoken from, exec is disabled if php is running as www-run or something that like, what it is as apache-module-handler. When switching the PHP handler to a fast-cgi version, PHP is only allowed to be invoked and running in the context of the user / owner of the account, what also is restricted in filesystem to only his own directory, etc.

Are there more aspects to put attention to?

[teppo]

It needs to be integrated into the right server configuration, I believe. On the servers I've spoken from, exec is disabled if php is running as www-run or something that like, what it is as apache-module-handler. When switching the PHP handler to a fast-cgi version, PHP is only allowed to be invoked and running in the context of the user / owner of the account, what also is restricted in filesystem to only his own directory, etc.

Are there more aspects to put attention too?

Without (hopefully) derailing this discussion too much, that's a good start: the server needs to be configured properly. Misconfigurations open it up for vulnerabilities. The most important thing in this context, though, is escaping commands and arguments properly, preferably using a whitelist approach. None of this applies to exec only, of course – code execution features just tend to increase the risks involved.

One thing that's kind of unique to code execution features is the amplification effect they have: if an attacker gains access to an application that doesn't allow her access to code, the most she can do is mangle your data. If there's a configuration option somewhere that allows her to control the output sent to exec(), such as the command it will execute, she suddenly has access to a lot more than just that.

Then again, the same thing goes for tools that allow you to edit your template files etc. online

[OllieMackJames]

Thanks all for your thoughts, we only raised 200$, and no one offered to code it

since I am not a coder, only a very happy user of the great processwire cms

I guess this will not happen - at least not now.

@horst question for you, could you perhaps include this in your great module croppable image?

Great discussion on the technical side of things, thanks for that!

[horst]

@horst question for you, could you perhaps include this in your great module croppable image?

What do you mean with "this"?

[Peter Knight]

Thanks all for your thoughts, we only raised 200$, and no one offered to code it

Everyone seems busy and I'm sure image compression will pop up again soon. I think it's a great start. 

Even if there was only a small declared monetary commitment and a high level technical discussion, I'm sure many more are interested as either backers, developers or designers.

BTW while I'd love to see image compression integration, what I'd really love to see is improved cropping options. And by that I mean more integrated first.

[OllieMackJames]

What do you mean with "this"?

@horst by this I mean image compression - preferably local on server

Don't know if this makes it clearer, as I am mainly focused on website speed results for usability reasons and for seo reasons.

thks

[Roope]

Hi all!

Just wanted to let you know that something wicked is coming your way:

https://github.com/BlowbackDesign/TinyPNG

It's still pretty much "work in progress" but currently module works and you're able to compress Pageimages by either manually with added tinify() method or automatically when image is resized using size() method. Croppable images can also be compressed by adding tinify() to the returned Pageimage object. On next stage I'll add support for (resized) embedded images at richtext editor and maybe some other features if needed. Time is little short right now but I'll try to make it happen sooner than later.

This module creates compressed images as a variation of original (or resized) image so any of the source files aren't overwritten. This ensures good quality and tries to avoid pixelated over compressed images. Module uses Tinify PHP client to connect TinyPNG API (and thus is available to use as is as an autoload module).

Please feel free to try out and tell me what you think!

[horst]

@Roope: this is good news.

I have only one suggestion:

I would avoid the uncontrolled automated mode! (for every size() request!) Instead I would bind it to a defined suffix option (tinify). Or you do it the opposite, check for a suffix (notinify) and compress all others. This is the only way to be able to also produce intermediate images wich resulting pageimages will be passed to another image->size() request.

Also, if the image is marked to get compressed, you need to check the setting for quality (if it can compress jpegs) and adjust it to 100%. If you compress images with quality settings lower than 100 and afterwards pass them to the compress service, you will get results with higher filesize! and with lower visual quality!

If you have any questions to my suggestion, I'll be glad to give more info and / or point to some.

https://processwire.com/talk/topic/6667-jpegoptimimage/#entry65531 (the first part)

http://images.pw.nogajski.de/jpegoptim/ (compared different imagesizer quality filesizes)

[Roope]

Thanks for the reply @horst! I was kind of hoping to get some feedback from you.

 

I fully get the point about issue with uncontrolled auto mode but I don't quite understand what you mean by binding compression to defined suffix option. Currently auto mode is hooked after Pageimage::size and (if enabled) compression is done when given size is allowed, type matches (png/jpeg) and API is available (client connected and monthly compression limit no exceeded). By adding custom image sizer option we could disable auto compression on cases where further image manipulation is done and then handle it manually last in chain or do you have something more sophisticated in mind?

$image->width(300, array('tinify'=>false))->pimLoad('prefix')->someManipulations()->pimSave()->tinify();

By the time it's also not prohibited to do re-compression to the images - maybe we should restrict such a use cases:

// tinify already tinifyed image
$page->image->tinify()->tinify();

// tinify resisized (and compressed) image when auto mode is enabled
$page->image->size(100,100)->tinify();

I was thinking about forcing (jpeg) quality to 100% but then I thought we don't need that since if one want's to compress images sized at lower quality, why to restrain that. But when you say lower quality actually makes higher filesize we sure need forced 100%. Since compression is done to existing pageimages, we can only force 100% when image is resized by PW. Do you think something like this could work or again, do you have something else in mind?

 

1. Hook before Pageimage::size - Check for compression and if ok, save filename to runtime variable.

2. Hook before ImageSizer::resize - Set pageimage quality to 100% if jpeg and filename matches to the one in runtime var.

3. Hook after Pageimage::size - Do the compression and return tiny variation of a pageimage.

 

Also I'd need your help (or maybe from @ryan) with resized images on rich text editors. First I thought hooking Pageimage::size would take care of this one also since it looks like that here in ProcessPageEditImageSelect::executeResize() but for some reason hook doesn't bite. Can you tell me why?

Edited April 21, 2016 by Roope

[horst]

Hi @Roope,
 
before we go into detail, I have one question about what PW version you want support: PW 2 or PW 3? If you are not rely on PW 2, there will be lot more possibilities. But there are ways for both versions.
 
What I suggest for both versions is, not to hook into Pageimage or that like, you only need and should hook into Imagesizer::resize. You can add any option to the options array of Pageimages. You can do this with individual API calls, or you can do this as a sitewide setting in the site/config.php under $config->imageSizerOptions.

$config->imageSizerOptions = array(
	'upscaling' => true,      // upscale if necessary to reach target size?
	'cropping' => true,       // crop if necessary to reach target size?
	'autoRotation' => true,   // automatically correct orientation?
	'sharpening' => 'soft',   // sharpening: none | soft | medium | strong
	'quality' => 90,          // quality: 1-100 where higher is better but bigger
	'hidpiQuality' => 60,     // Same as above quality setting, but specific to hidpi images
	'defaultGamma' => 2.2,    // defaultGamma: 0.5 to 4.0 or -1 to disable gamma correction (default=2.0)

	'tinify' => true,      // your option for all imageformats here

// or your options for specific formats
	'tinifypng' => true,
	'tinifyjpg' => true,
	'tinifygif' => false,
// or whatever you like ...
	);

Regardless if you use sitewide or individual options, they will make their way into the ImageSizer. Using sitewide options and overwriting them with an individual value passed as $options array in the API works too. This is all supported in PW since version 2.5.11.

The only thing what I would hook in, (if you need to support PW2 too), is the Imagesizer::resize method:

• before: check if tinify is requested and if true, adjust quality for jpegs to 100%, (and, maybe, add "tiny" to the suffixes array ??)
• after: pick up the resized image variation and transfer it to your tinify service

If you can go with a PW 3 version only, I have some thoughts to standardize optimizations like yours: they should be able to register themself into the ImageRenderingChain as optimization / compression tools. This way, the PW ImageEngines would take care for most of the above stuff and simply call (at last step) your provided service.

----

// example for PW 2.5.11+

    public function init() {
        $this->pages->addHookBefore('ImageSizer::resize', $this, 'hookEventBeforeResize');
        $this->pages->addHookAfter('ImageSizer::resize', $this, 'hookEventAfterResize');
    }

    public function hookEventBeforeResize($event) {
        $this->image = $image = $event->object->image;
        // first do a check if tinify is requested
        if(!$image->tinify) {  // or do we need to look into $image->options['tinify'] ?? for user options ??
            $this->tinify = false;
            $event->replace = false;
            return;        
        }
        // tinify is requested, set a flag for the hookafter and adjust quality
        $this->tinify = true;
        $this->image->quality = 100;
    }
        
    public function hookEventAfterResize($event) {
        // was tinify requested or not
        if(!$this->tinify)
            $event->replace = false;
            return;
        }
        // pick up the variation file and pass it to the tinify service
        ...
    }

Edited April 21, 2016 by horst
added a code example for PW 2.5.11+

[Roope]

Thanks for the detailed info @horst! Much appreciated.

 

Actually the first implementation of this was hooked to ImageSizer::resize but then I decided to abandon that route since while it was doing the actual compression alright, I was having hard time to figure out how to return tiny variation instead of uncompressed resized original from $image->size() call. That's why I chose hooking straight to Pageimage::size where it is easy to just replace return with tinifyed image. That said, I still can't see how this could be accomplished working only with ImageSizer::resize hooks (without overwriting the original resized source). How to tell PW we wan't to use tiny variation of a variation without hooking Pageimage::size?

 

I definitely also want to support compressing original unresized images so Pageimage::tinify method is there to handle such a use cases (also serves us when auto mode is not preferred option).

 

The point you gave here that with ImageSizer::resize we're able to use and modify $config->imageSizerOptions makes it clear that this is the right place to hook for compression. So what do you think if the routine would go like (simplified):

1. Hook before ImageSizer::resize
   Check that tinify is not disabled at image options and compression can be done (API and stuff). Then set tinify flag and quality to 100%.
2. Hook after ImageSizer::resize
   If tinify flag is set, copy and compress the image and save it to the page as tiny variation.
3. Hook after Pageimage::size
   Check for tiny variation and return that if found.

So basicly it would be just like your example for PW 2.5.11+ but Pageimage::size still hangs around to return tiny variation. Do you see any downside or is there still something I'm not getting?

 

What goes for PW support, I wouldn't like this to be PW3 only. Specially right now when official stable is still PW2 and anyway it's going to stick around with us way longer. What I would like is to have at least one good solid release available for the PW2 - where support for 2.5.11 and up is reasonable enough. Then we're good to go ahead towards PW3 and make use of all the beauty that it brings (like ImageRenderingChains and stuff which I know none about).

 

P.S. Forget what I wrote on previous post about ProcessPageEditImageSelect::executeResize() - it was just me tripping, not the PW. It works fine.

[horst]

The point you gave here that with ImageSizer::resize we're able to use and modify $config->imageSizerOptions makes it clear that this is the right place to hook for compression. So what do you think if the routine would go like (simplified):

• Hook before ImageSizer::resize

  Check that tinify is not disabled at image options and compression can be done (API and stuff). Then set tinify flag and quality to 100%.

• Hook after ImageSizer::resize

  If tinify flag is set, copy and compress the image and save it to the page as tiny variation.

• Hook after Pageimage::size

  Check for tiny variation and return that if found.

So basicly it would be just like your example for PW 2.5.11+ but Pageimage::size still hangs around to return tiny variation. Do you see any downside or is there still something I'm not getting?

Yes, this is correct, and there should be no need to hook into other parts, I think.

[Roope]

@horst - I'd need your help bit further with this one...

When hooking to ImageSizer::resize is there a way to get the original resized Pageimage object? Your code example below had something like this:

public function hookEventBeforeResize($event) {
   $this->image = $image = $event->object->image;
}

But $event->object->image is referring to the array that contains information about the image width and height, not the actual Pageimage.

Also, as a reference if someone else is using info from this thread, on ImageSizer you can't set quality property directly - it'll throw fatal error (cannot access protected property ImageSizer::$quality). Instead there is a setter method for doing this:

public function hookEventBeforeResize($event) {
   $image = $event->object;
   $image->setQuality(100);
}

Why I need Pageimage object from ImageSizer is that it would make returning compressed tiny variation a breeze when you can just clone the source Pageimage:

private function tinyPageimage(Pageimage $image) {
   $tinyimage = null;
   $filename = $this->tinyFilename($image);
   
   // tiny image file exists, let's clone the original Pageimage
   if(is_file($filename) && filemtime($filename) >= filemtime($image->filename())) {
      $tinyimage = clone $image;
   }
   
   // tiny image file is not available, proceed if we can compress the source
   if(!$tinyimage && $this->compressionAllowed()) {
      // copy the original source and clone it if compression was ok
      if(copy($image->filename(), $filename)) {
         if($this->compressImage($filename) !== true) unlink($filename);
         else $tinyimage = clone $image;
      } else {
         throw new WireException("Failed to copy Tiny image variation: $filename");
      }
   }
   
   // return tiny variation of Pageimage when available
   if($tinyimage instanceof Pageimage) {
      $original = $image->original ? : $image;
      $tinyimage->setFilename($filename);
      $tinyimage->setOriginal($original);
      return $tinyimage;
   }
   
   return $image;
}