Issues with image resize and CropImage module

[MadeMyDay]

I put this topic here, because I am not sure if it is matter of the image API or a problem with Apeisa's CropImage module or both.

Sometimes clients try to upload an image which is too big for image resizing. This is not a bigger problem if this would just fail. But I've experienced some strange behaviours:

1. Problem:

Upload works (progress bar ends at 100%) but thumbnail is not generated for the image field (perhaps not too large for upload but too large for resize). This ends up in an uploaded original image in the assets/files folder, but also in a copy of that original renamed to myimage.0x100.jpg which has the same size. If the thumbnal isn't generated the image remains in the files folder. Since it is not written to the database it is not visible after page save. So after editing that page again and another upload the problem repeats.

1. Possible solution:

If the thumbnail generation fails, delete both, the uploaded original and the copy and give feedback that something failed.

2. Problem

Upload works and thumbnail is generated. But then Apeisa's module fails to generate the defined CropImages (perhaps because generation of three different sizes is too much for a bigger image). Then there is no feedback, the page can be saved. After opening it again the page completely fails with a server error. No possibility to fix the problem without deleting the entry from that image from the database directly.

2. Possible solution:

If CropImage generation fails, give feedback and ... well. What then? I think the only way would be to delete the original as well, because otherwise the problem repeats every time after page save.

General solution for the image / cropImage field:

Possiblity to define a max size (in pixels as well as in kB/MB). Then check the size after upload and deny all image resizing when the image is bigger/larger than defined. I think this could solve a lot of problems, especially on weak servers.

[horst]

Hi MadeMyDay,

you can define allready a max size for uploaded images in the admin for the (crop)-images field under setup fields (your image field) input:

Optionally enter the max width and/or height of uploaded images. If specified, images will be resized at upload time when they exceed either the max width or height. The resize is performed at upload time, and thus does not affect any images in the system, or images added via the API.

This way there is only one resize with the big image version. Following resizes are faster.

EDIT:

Additionally you may try to bump up the allowed memory usage for php to 128MB or 256MB:

in htaccess: php_value memory_limit 128M or 256MB

Edited August 21, 2013 by horst

[MadeMyDay]

Hi Horst,

I am aware of that setting. But the problem is the same. If the uploaded image is too big to resize, that resize ends up in the same behavior as described under 1.)

edit: The resolution as only setting is not enough by the way. There are images with 3000x2000 pixels which only have 500kb. But there are also ones with 1024x768 with > 1MB.

[horst]

@MadeMyDay: ok, you may look here and try it if it solves your issues too: http://processwire.com/talk/topic/3718-lots-of-images-to-resize-timeout-in-frontend/?p=36291 (this post and above)

Also have edited first post! (php memory usage!)

[owzim]

Hi Horst,

I am aware of that setting. But the problem is the same. If the uploaded image is too big to resize, that resize ends up in the same behavior as described under 1.)

edit: The resolution as only setting is not enough by the way. There are images with 3000x2000 pixels which only have 500kb. But there are also ones with 1024x768 with > 1MB.

It does not matter how large the initial filesize is because if the image is cropped, or edited in any other way, all the pixels go (of course uncompressed) into ram and 3000x2000 are a LOT of pixels. 

I had the same problem with the 100% progress bar, and the php_value memory_limit setting solved it.

Edit: I still would consider training the clients to resize their images before uploading. I think if they work with content they cannot assume that any system eats any type or amount of data. You have to preprocess everything in some way, may it be text or images.

[MadeMyDay]

Yes, all of your solutions are possible solutions. But nevertheless would it be nice, if PW would allow to a.) restrict uploads to size/resolution and b.) give feedback if something fails. I also had the problem that my dev server processed all without problems and on the client's hosted solution everything failed without the possibility to raise the memory limit. And the biggest problem was the produced copy of the original image. Because it wasn't deleted I had background images with > 5 MB without noticing because the API found the image "myimage.1920x0.jpg" and thought everything is fine.

I still would consider training the clients to resize their images before uploading. I think if they work with content they cannot assume that any system eats any type or amount of data. You have to preprocess everything in some way, may it be text or images.

Of course. But you know how it is, they will ignore it. And if the site in the admin gets blank it is a big problem. So a solution which reminds them not to upload too big pictures would be nice.

ok, you may look here and try it if it solves your issues too: http://processwire.c...ontend/?p=36291 (this post and above)

I will take a look, thanks!

[horst]

@MadeMyDay: I think filesize doesn't matter, like owzim said: important is the needed RAM for the uncompressed image.
 
@owzim: I think you are right (to 95%), but the timeout can matter too.
 

EDIT:

@MadeMyDay:

So a solution which reminds them not to upload too big pictures would be nice.

PHP: settings for post_max_size & upload_max_filesize can be an option!

(If the customers / clients and their coworkers ignore advise, you have to push them harder!)

Edited August 21, 2013 by horst

[horst]

 It does not matter how large the initial filesize is because if the image is cropped, or edited in any other way, all the pixels go (of course uncompressed) into ram and 3000x2000 are a LOT of pixels.

Yeah, I remember back 15 years or some more, - we have compressed very large images with jpeg quality 1 to a minimum filesize and have send it per Email to recipients where the needed RAM of the uncompressed image was higher than their available system memory!

[interrobang]

Maybe you can modify Somas ImageMinSize.module http://processwire.com/talk/topic/4091-imageminsize/ to remove the image if its dimensions exceed your allowed dimensions.

[MadeMyDay]

Good idea! Will try that.

[Soma]

post_max_size and upload_max_size isn't an option as it wouldn't allow for multiple files and also upload would fail silently.

But there's a maxFileSize on file fields, it's not I'm not sure why it's not in the field settings. 

... else if($this->maxFileSize > 0 && $size > $this->maxFileSize) { ...

I guess it can be set via a hook and you could try setting it to a reasonable size.

Sorry, actually it's in teh WireUpload class and not the file field. But anyway it's there for a reason. And it's also maxFilesize in the file inputfield, but it checks for the php post_max_size to get the max size, and this is for all files in a multiple file field.

[ryan]

Can you guys tell if the issue is in the core or with CropImage? If a large image is causing it to fail and leave an untracked file, or something broken, then that's something we'd want to fix. Though when an image is so large that it consumes all memory and goes fatal, I'm not positive if the usual error checking will work. So the option of defining some reasonable maximums, as mentioned above, seems like a good idea. 

[horst]

So the option of defining some reasonable maximums, as mentioned above, seems like a good idea.

It is possible to define a maximum number of allowed pixel (like Megapixel) somehow and it can be checked before loading the imagefile into memory with the getimagesize function:

$size = getimagesize($filename);
$maxPix = $size[0] * $size[1];

[ryan]

Horst, this sounds like a good way to go. As the resident image expert–what would you suggest is a good maximum for us to start with in a /site/config.php setting? I'm thinking we probably have to set something that would be compatible with a PHP 32M memory_limit setting. Though not positive we can really identify an exact number without trial and error. I was thinking of starting with 15,000,000 pixels (5000x3000), though even that may be too large. 

[horst]

Ryan, wouldn't it be better to check if the current available memory is enough for a image resize?

If we restrict it in the config.php for 32M, all above that (64M, 128M, 256M, ...) have to change the setting. And if they don't know it, they first run into the limitation.

I think if we try to find a point for the maximum we have to guess how much RAM PW has allocated before the call to image resize.

I don't know how much memory PW allocated for a 'normal' ? request, but lets say it would be something like 14M for example, we have 18M for the image resize.

We need memory for the source image and for the target image plus memory for the image operation(s). - lets use a factor 2.5

18 / factor 2.5  = 7.200.000 / 3 (channels RGB) = 2.400.000 (this would be the setting for the config.php)!

This could be a image of 1250 x 1920 px.

But it will crash if PW has allocated 15M and not 14M.

That said, one possible way (without a setting in config.php) could be to check the available memory:

function checkMemoryForImage($imageFilename) {
	// images pixel
	$size = getimagesize($imageFilename);
	$imgPix = $size[0] * $size[1];
	// read PHP maxMemory from php.ini
	$sMem = trim(ini_get('memory_limit'));
	preg_match('/^[0-9].*([k|K|m|M|g|G])$/', $sMem, $match);
	$char = isset($match[1]) ? $match[1] : '';
	switch(strtoupper($char)) {
		case 'G':
			$maxMem = intval(str_replace(array('G','g'),'',$sMem)) * 1073741824;
			break;
		case 'M':
			$maxMem = intval(str_replace(array('M','m'),'',$sMem)) * 1048576;
			break;
		case 'K':
			$maxMem = intval(str_replace(array('K','k'),'',$sMem)) * 1024;
			break;
		default:
			$maxMem = intval($sMem);
	}
	// read current allocated memory
	$curMem = memory_get_usage(true);  // memory_get_usage() always is available with PHP since 5.2.1
	// check if we have enough RAM to resize the image
	if($maxMem - $curMem > 2.5 * $imgPix) {
		return true;
	}
	return false;
}

The code is meant as example. There should be only one call per Request (or per Session?) to the php.ini to get the maxMem, and not one call per image resize, etc.

[MadeMyDay]

Sounds brilliant!

[horst]

In the above code I have forgotten the channels, so the correct code could be:

function getMaxMem() {
	// read PHP maxMemory from php.ini
	$sMem = trim(ini_get('memory_limit'));
	preg_match('/^[0-9].*([k|K|m|M|g|G])$/', $sMem, $match);
	$char = isset($match[1]) ? $match[1] : '';
	switch(strtoupper($char)) {
		case 'G':
			$maxMem = intval(str_replace(array('G','g'),'',$sMem)) * 1073741824;
			break;
		case 'M':
			$maxMem = intval(str_replace(array('M','m'),'',$sMem)) * 1048576;
			break;
		case 'K':
			$maxMem = intval(str_replace(array('K','k'),'',$sMem)) * 1024;
			break;
		default:
			$maxMem = intval($sMem);
	}
	return $maxMem;
}

function checkMemoryForImage($imageFilename, $maxMem) {
	// images pixel
	$size = getimagesize($imageFilename);
	$channels = isset($size['channels']) && $size['channels']>0 && $size['channels']<=4 ? $size['channels'] : 3;
	$imgMem = $size[0] * $size[1] * $channels;
	// read current allocated memory
	$curMem = memory_get_usage(true);  // memory_get_usage() is always available with PHP since 5.2.1
	// check if we have enough RAM to resize the image
	return ($maxMem - $curMem > 2.5 * $imgMem) ? true : false;
}

$maxMem = getMaxMem();

foreach($imageFiles as $imageFile) {
	if(!checkMemoryForImage($imageFile, $maxMem)) {
		throw new WireException("Cannot load the imagefile {$imageFile->name} into Memory! Imagedimensions are to large.");
		// or e.g. with modules use $this->error("Cannot load the imagefile {$imageFile->name} into Memory! Imagedimensions are to large.")
		continue;
	}
	// do the resize
	...
}

[Martijn Geerts]

@horst, I really like this way of thinkin' !

[ryan]

Nice work Horst!