memory_limit error with images

[Spica]

I have a lot of image heavy pages. So I have several memory_limit errors with resizing.

I wonder, if pw's image resizing functionality has any routine to avoid breaking the whole page.

If not, would it be enough reading the memory size and compare it to the calculated target image size before starting the resizing?

[Martijn Geerts]

What PW version do you use?

[Spica]

2.6.21 dev

[Martijn Geerts]

Think Horst or someone who is more knowledgeable then me will respond soon.

[Tom.]

I've come across this problem recently too. I wanted to resize 3000 x [x] to 1920 x [x] but it wanted over 64MB to do that resize. I just increased the memory_limit to 254MB. 

I have no idea what's "normal" memory usage for an image resize of those large dimensions. I'm using PW3.0.5.

[Spica]

I cant increase memory limit. Anyway, there would be still a limit, that could be reached and shouldnt lead to break the page. I think the needed memory size can be calculated from the origin image, but also depends on how many instances get done at once, generally speaking how the script handles the memory resources. But as I am not realy deep into that I better ask.

[LostKobrakai]

Image resizing does need to load the whole file into the ram memory, which means the image's data cannot benefit from any compression, like saved images do. That's why images need way more space to be resized than their raw file size would suggest.

[Spica]

yes, lostkobrakai. Size can be calculated like here

But am I done with a simple size calculation of the file and checking it against the memory limit? Or are there other processes when I start a resizing that take additional memory.

[horst]

Hi @Spica,

please have a look into ImageSizer. PW already calculates dimensions and compare the needed amount of ram with available ram _before_ opening an image or _before_ creating another instance of an already loaded image:

• https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/core/ImageSizer.php#L412
• https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/core/ImageSizer.php#L1913

Only thing what comes to my mind is to increase memory, use smaller images or upload only one after the other and not a lot in parallel. Have you tried to upload multiple images as zip-archive?

-------------------------------------------------------------------------------------

You can test and call the method from the outsite of imagesizer too, if you like:

$info = getimagesize($image->filename);
$sourceDimensions = array($info[0], $info[1], $info['channels']);
$result = ImageSizer::checkMemoryForImage($sourceDimensions);
var_dump($result);

EDIT: the only thing what maybe happen in your system is that PHP isn't able to read a setting for max_mem, please have a look here to this line: https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/core/ImageSizer.php#L1934

If you use the above code for a test, and var_dump($result); will show null, PHP isn't able to read a seting for max_mem or there isn't one set. In this case, please also refer to output from phpinfo().

Edited January 25, 2016 by horst

[Spica]

Thanks Horst, for the explanation.

if I put your snippet into my template I do get bool(true) bool(true) as output, so the memory limit should be read.

But when I set my output dimension of the image up I still get:

Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 18700001 bytes) (line 1655 of /usr/www/users/user/ProcessWire/wire/core/ImageSizer.php)

Dont know why line 1655 causes the error, as I am only resizing with image->width and image->height

[horst]

Hi Spica, thanks for the report. With todays large images from smartphones, 128MB is to small for them to be processed in a good manner.

The line 1655 is image sharpening! There is no extra checking for memorylimit in this function. Please can you test the same page / image with the error and than add this additional option to the options array of the api call:

$image->size($width, $height, array('sharpening' => 'none'));

or alternatively, set the sharpening to 'none' in the site/config.php

$config->imageSizerOptions = array(
	'upscaling' => true,     // upscale if necessary to reach target size?
	'cropping' => true,      // crop if necessary to reach target size?
        'autoRotation' => true,  // automatically correct orientation?
    'sharpening' => 'none',  // sharpening: none | soft | medium | strong
	'quality' => 90,         // quality: 1-100 where higher is better but bigger
	'hidpiQuality' => 60,    // Same as above quality setting, but specific to hidpi images
	'defaultGamma' => 2.0,   // defaultGamma: 0.5 to 4.0 or -1 to disable gamma correction (default=2.0)
	);

[horst]

As sharpening is the last manipulation step, we can try to optimize the imagesizer in that regard that we try to free memory for one image instance before we pass the $thumb-instance to the sharpening method.

If you like, you can first backup the imagesizer.php and then rearrange this part

		// optionally apply sharpening to the final thumb
		if($this->sharpening && $this->sharpening != 'none') { // @horst
			if(IMAGETYPE_PNG != $this->imageType || ! $this->hasAlphaChannel()) {
				// is needed for the USM sharpening function to calculate the best sharpening params
				$this->usmValue = $this->calculateUSMfactor($targetWidth, $targetHeight);
				$thumb = $this->imSharpen($thumb, $this->sharpening);
			}
		}
		// write to file
		$result = false;
		switch($this->imageType) {
			case IMAGETYPE_GIF:
				// correct gamma from linearized 1.0 back to 2.0
				$this->gammaCorrection($thumb, false);
				$result = imagegif($thumb, $dest); 
				break;
			case IMAGETYPE_PNG: 
				if(!$this->hasAlphaChannel()) $this->gammaCorrection($thumb, false);
				// always use highest compression level for PNG (9) per @horst
				$result = imagepng($thumb, $dest, 9);
				break;
			case IMAGETYPE_JPEG:
				// correct gamma from linearized 1.0 back to 2.0
				$this->gammaCorrection($thumb, false);
				$result = imagejpeg($thumb, $dest, $this->quality); 
				break;
		}
		if(isset($image) && is_resource($image)) @imagedestroy($image); // @horst
		if(isset($thumb) && is_resource($thumb)) @imagedestroy($thumb);
		if(isset($thumb2) && is_resource($thumb2)) @imagedestroy($thumb2);
		
		if(isset($image)) $image = null;
		if(isset($thumb)) $thumb = null;
		if(isset($thumb2)) $thumb2 = null;

to become this one:

                if(isset($image) && is_resource($image)) @imagedestroy($image);
                if(isset($thumb2) && is_resource($thumb2)) @imagedestroy($thumb2);
                if(isset($image)) $image = null;
                if(isset($thumb2)) $thumb2 = null;

		// optionally apply sharpening to the final thumb
		if($this->sharpening && $this->sharpening != 'none') { // @horst
			if(IMAGETYPE_PNG != $this->imageType || ! $this->hasAlphaChannel()) {
				// is needed for the USM sharpening function to calculate the best sharpening params
				$this->usmValue = $this->calculateUSMfactor($targetWidth, $targetHeight);
				$thumb = $this->imSharpen($thumb, $this->sharpening);
			}
		}
		// write to file
		$result = false;
		switch($this->imageType) {
			case IMAGETYPE_GIF:
				// correct gamma from linearized 1.0 back to 2.0
				$this->gammaCorrection($thumb, false);
				$result = imagegif($thumb, $dest); 
				break;
			case IMAGETYPE_PNG: 
				if(!$this->hasAlphaChannel()) $this->gammaCorrection($thumb, false);
				// always use highest compression level for PNG (9) per @horst
				$result = imagepng($thumb, $dest, 9);
				break;
			case IMAGETYPE_JPEG:
				// correct gamma from linearized 1.0 back to 2.0
				$this->gammaCorrection($thumb, false);
				$result = imagejpeg($thumb, $dest, $this->quality); 
				break;
		}
		if(isset($thumb) && is_resource($thumb)) @imagedestroy($thumb);
                if(isset($thumb)) $thumb = null;

We only have to move 4 lines that try to release gd image handles above the call to the sharpening method.

[Spica]

I tested like this

my target picture (thumb) is 2500x1870

I tested two memory limits, 128 and 64

with array('sharpening' => 'none')

both tests rendered the image

with sharpening on

64M

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 10000 bytes) in /Applications/MAMP/htdocs/ProcessWire-dev/wire/core/ImageSizer.php on line 1632

128M

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 18700001 bytes) in /Applications/MAMP/htdocs/ProcessWire-dev/wire/core/ImageSizer.php on line 1655

then with your imageSizer.php modification and sharpening on

128M
image gets rendered

64M
 

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 10000 bytes) in /Applications/MAMP/htdocs/ProcessWire-dev/wire/core/ImageSizer.php on line 1678

So it seems to me, that there are several more instances created for different image effects which take the memory uncontrolled. Even if 64M may seem to be a very low memory to someone, but as devices resolution may rise in future high resolution images even higher memory limits may get broken.

[LostKobrakai]

Even if 64M may seem to be a very low memory to someone, but as devices resolution may rise in future high resolution images even higher memory limits may get broken.

That's a problem nobody here can really solve. You surely wouldn't expect an old amiga to be able to process todays images. Optimizing image processing further down the stack may be possible, but the code we control can only make sure to prevent any unneccessary parallel memory usage and that's about it.

[Spica]

Sorry, I was unclear. I did not mean to optimize for 64M. It is just a limit, as 512M also is. And 512 can be easily reached and lead to break the page when we use large images and lots of instances for effects. I can not come up with a solution as I am not realy into the code. I have no clue how often the instances are needed or how to deal best with it.

My consideration is: What environment is needed to not get broken pages? Is it predictable? If I now set the limit to 256 or even 512, does it guarantee to not cause any error? If these errors can not be avoided by the script, how to calculate the max. needed limit (therefore we should know how many instances are needed for what effect)?

[horst]

  So it seems to me, that there are several more instances created for different image effects which take the memory uncontrolled. Even if 64M may seem to be a very low memory to someone, but as devices resolution may rise in future high resolution images even higher memory limits may get broken.

Thanks for testing Spica!

All what you have found out belongs to sharpening, and indeed there are two instances where currently no memory-check is invoked. Those both (not several!) instances belong to the unsharpMask function, what is enabled by default to create the best possible result for your images. So, as we have to find a balance between fast image processing and to avoid going out of memory, every user should take a bit care with his sites. If you are low on memory, you definetly should limit the max dimensions for upload images.

Besides that, you uncovered that we need to describe more features we have under the hood with image processing. With a slide change in the $config settings we can disable the usage of the UnsharpMask for sharpening and switch to another algorithm, what needs much much lesser memory. This unsharp algorythm was there earlier, but we decided to switch to the visually better working USM by default in times where 256M seems to be a common value for memory usage.

Conclusion: we will implement memory checking for those two instances currently not covered too, and we will provide flags within the $config settings that enables users to simply switch between sharpening algorythms in systems with low memory.

Once again, thanks for your tests and reports, it is much appreciated.

Edited January 27, 2016 by horst

[Spica]

That sounds good.

every user should take a bit care with his sites

That's right. Would you please provide some information on calculation basics. E.g. does the unsharpen mask just need an own instance, is it calculabe at all, is it only the unsharpen mask that increases the memory need.

So one can finde a balance between using the effect, the thumbs size and the memory limit.

[horst]

I have a test version of imagesizer now where the chosen algorithm depends on available memory. If there is enough memory, it uses USM, if there is less memory, it levels down to use the other algorithm, and if there also isn't enough memory available for this, sharpening is completly skipped.

After some testing I will commit this to github.

[horst]

It's on the way: https://github.com/ryancramerdesign/ProcessWire/pull/1646

[noodles]

Just hopping in on this topic. 

I struggle with a webspace that has a 128M limit and I am trying to upload a picture with 2848x4272 pixels.

The upload got stuck at 100%. Crawling through the console and error logs, I found out that it is a memory limit issue. PW would have liked 2MB more. ;-)

It's running on PW 3.0.11 and the error leads back to ImageSizerEngineGD.php:294

	protected function imRotate($im, $degree) {
		$degree = (is_float($degree) || is_int($degree)) && $degree > -361 && $degree < 361 ? $degree : false;
		if($degree === false) return $im;
		if(in_array($degree, array(-360, 0, 360))) return $im;
		return @imagerotate($im, $degree, imagecolorallocate($im, 0, 0, 0));
	}

I just set autoRotate to false and now the upload works.

Is this a miscalculation? Any recommendations for a proper workaround, like rotating afterwards, when the upload is finish? I don't know if this could solve this problem.

Tried with both, a "regular" Image field and CroppableImage (which I still love, thanks Horst! )

[horst]

 I struggle with a webspace that has a 128M limit and I am trying to upload a picture with 2848x4272 pixels.

The upload got stuck at 100%. Crawling through the console and error logs, I found out that it is a memory limit issue. PW would have liked 2MB more. ;-)

It's running on PW 3.0.11 and the error leads back to ImageSizerEngineGD.php:294

	protected function imRotate($im, $degree) {
		$degree = (is_float($degree) || is_int($degree)) && $degree > -361 && $degree < 361 ? $degree : false;
		if($degree === false) return $im;
		if(in_array($degree, array(-360, 0, 360))) return $im;
		return @imagerotate($im, $degree, imagecolorallocate($im, 0, 0, 0));
	}

I just set autoRotate to false and now the upload works.

Is this a miscalculation? Any recommendations for a proper workaround, like rotating afterwards, when the upload is finish? I don't know if this could solve this problem.

The image craetion / rotation belongs to the creation of an AdminThumnail, I believe. It has nothing to do with the upload itself. Or do you have defined any max-dimension with the imagesfield?

But if you are on PW3, why don't you use another image engine than GD? If you can go with IMagick or NetpbmCLI or ImageMagickCLI, yo will not run into those problems. The GD-lib is the only one engine that always need to load the full uncompressed image into memory as one block. The other engines are able to process images in lines or chunks.

Edited April 25, 2016 by horst

[noodles]

Hi Horst,

thank you for your instant feedback. My server doesn't have ImageMagick, so I uploaded the executables and installed your ImageSizerEngineIMagickCLI module.

However, the error still routes to the ImageSizerEngineGD.php

Do I have to configure something elsewhere? Sorry if this is a dumb question.

Oh, and disabling the thumbnails didn't help. No max dimensions set.

[horst]

... My server doesn't have ImageMagick, so I uploaded the executables and installed your ImageSizerEngineIMagickCLI module.

However, the error still routes to the ImageSizerEngineGD.php

You need to be able to make the ImageMagick executable in your system. If they are installed correctly, one can call them via php function exec().

You need to add the full featured directory path of the executable in the module.

If PW cannot find it or PHP isn't able to execute it, PW automatically passes the image rendering to the GD-lib.

1) So, first you need to check if your ImageMgick executable is registered / installed correctly in your system.

2) can php call and use it?

3) is the right path defined in PW? (in ImageSizerEngineIMagickCLI module)

If all would be true, the ImageSizerEngineGD would not be invoked.