max file size validation for file/image fields

[chrizz]

Are there any plans to add a client side file size validation for file and image fields? Would it be a possible way to use the FILE API to check the size before uploading? Using max dimensions is not useful at all imho.

Or - if not - how do you handle the case that clients can upload large images (large = means file size) which break the PHP memory limit on a shared webspace?

 

 

[LostKobrakai]

I recently looked into using chunked uploads to allow for file uploads beyond php limits. What I found is that it's not really simple to get the "actual" max upload filesize of php (more here). Also the max-dimensions are not really meant to prevent too big files.

Anyway the results of my dive into chunked uploads can be found here: 
https://gist.github.com/LostKobrakai/ddb11c64dcb0f1bf1c8eed4dd534a868

I'll try to implement it in the core as soon as I find some more time, but currently it's tough to get to any open-source work.

[chrizz]

@LostKobrakai thanks for having a look! But to make sure (and avoid extra work) I was not talking about chunked uploads but only about a file size limit for a single file. 

Uploading a 5MB file (4000x4000px) works fine on the shared webspace but when the ImageResizer enters the game the PHP memory limit is hit when it's trying to resize the image to 1000x1000px

To avoid this we need to optimize the file size before uploading it to PW. Having a 2MB 4000x4000px image works fine when it is resized.

In my imagination the file/image field get a new option: max file size. The validation (simple JS check if the file selected for uploading is larger than the limit) happens on client side with the FileReader

 

[LostKobrakai]

That's just not how images work. You optimize the image size by using compression methods. But to manipulate (scale, …) an image it has to be decompressed first, so your optimization should in theory do next to nothing in terms of memory footprint. Also @horst already did his best to improve the image sizing tools as far as possible to detect memory exhaustion before actually hitting the limit. But there are just limits to what's detectable beforehand. 

P.S.: It's not advices to optimize source images at all, because the quality of thumbs will suffer (more than the compression did your source image)

[chrizz]

well... in theory... ^^

I just tested it:

• jpeg
• 3888 x 2592
• original file size: 5,1 MB
• ==> hits the PHP limit

same image after saving it again with only 50% quality

• jpeg
• 3888 x 2592
• new file size: 0,8 MB
• ==> does not hit the PHP limit, scaling works as expected

I expected these results because reducing the quality does not reduce the actual size of the image but the number of information (e.g. in the first example every pixel has it's own color definition while in the second example color information is combined e.g. 4 pixels contain the same color ==> file size is reduced)

but maybe @horst is deep into the theory and might be able to explain how it really works  

[horst]

@chrizz your calculation is totally wrong. Only thing that is of interest for imagesizer is uncompressed image in memory. You do not hit a filesize limit, you hit a memory limit.

Why you run into it, I can't say. But I can say that we do runtime checks for available memory before opening an image in memory, but also we use a fast calculation depending on most common images for that. Without seeing your original images, I guess your original image has some image markers like EXIF, IPTC, VENDOR-Tags etc. what may be a little over the top of what we use for our precalculation. Theoretically it is possible to pack a lot of data into this, but typically it isn't that big. Maybe an edgecase? (Calculation like we do, fits just into available runtime memory, but without the ImageMarkers?)

But one thing is a physical fact: FilesizeCompression cannot reduce needed memory limit.

(and therefor I do not need deep theory, this is simply how computers work since ever. In my earlier days, around 1994, we sometimes do some jokes: we compressed big dimensioned jpeg images with the lowest quality setting / highest compression rate, and send them as previews to Artdirectors with a question to them. Always when they tried to display it, it got loaded / uncompressed into memory and their systems imediately got freezed / unusable. So, we only have done that to very unfriendly people who really deserverd a little shock. )

So, what ever you have changed in your second saved imagefile, what is responsible for the fit into memory than, it is not filesize compression.

A serious test would need to log the runtime available memory for both images before trying to open it, and the same after they allocated the memory, and compare all that.

Also a full check of all file parts in the imagefile (markers and image data). Without that, it says null to nothing and only opens room for glasball looking, guessing or misinterpretions.

[LostKobrakai]

Sure a low quality image is better compressable (lots of same color values), but it's still 3888 x 2592 x 8bit x 4 (rgb + alpha) to just hold the image in memory. Maybe some other operation in the proces is memory intensive. 

[chrizz]

here they are. Didn't find any relevant meta data in the image itself. It's imho only about the quality of the jpeg (97 vs. 50). The first one hits the memory limit, the second one didn't.

 

@horst edited  

examples.zip

Edited February 20, 2017 by chrizz

[horst]

not uploaded through the forums software, the @Real_Original_Images please. packed into a zip should work.

[chrizz]

I quickly wanted to follow up this issue. I am not sure if edition a post including a mention triggers a notification

[Mikie]

Rather than starting a new thread, I'd like to bump this as I am after the same thing.

My use case is that I have some file fields that accept html5 video files (webm / mp4) for short video loops that I'd ideally like to be under 2mb, but I can't stop the client from being lazy and uploading much larger files anyway. 

I'm going to write a module to achieve this restriction for now, but I was surprised it wasn't available out of the box. 

Thanks!

[Robin S]

@mke, there is a property and helper method in InputfieldFile that allows a maximum files size to be set but there is no config field for it in admin and you would need to set the max file size in a hook in /site/ready.php:

$wire->addHookBefore('InputfieldFile::render', function(HookEvent $event) {
    /* @var InputfieldFile $inputfield */
    $inputfield = $event->object;
    if($inputfield->hasField == 'YOUR_FIELD_NAME') {
        $inputfield->setMaxFilesize('2m'); // Set the max file size in bytes or use string like "30m", "2g" "500k"
    }
});

Also, for some reason max file size settings of less than 2MB are not enforced - see here.

[abdus]

Another alternative to enforce arbitrary filesize limitation is using File API. This has the advantage of preventing file uploads before even they hit to the server, saving bandwidth and resources.

Given a form like this

<form>
    <p data-maxsize-message>Files larger than {MAX_SIZE} aren't allowed</p>
    <input type="file" name="uploadables" multiple><br>
    <div data-error-wrapper></div>
    <button type="submit" name="submit" value="submit">Submit</button>
</form>

You can check file sizes, show and error and prevent file uploads using something like this:

const MAX_SIZE = 0.2e6; // bytes => ~0.2MB
const MAX_SIZE_STR = '0.2MB';

const form = document.querySelector('form');
const fileInputs = form.querySelectorAll('[type=file]', form);
const errorWrapper = form.querySelector('[data-error-wrapper]');
const submitButton = form.querySelector('[type=submit]');
const maxSizeMessage = form.querySelector('[data-maxsize-message]');

// Display max allowable file size
maxSizeMessage.innerText = maxSizeMessage.innerText
    .replace('{MAX_SIZE}', MAX_SIZE_STR);

// listen to all file inputs for new files
[].slice.call(fileInputs)
    .forEach(inp => inp.addEventListener('change', handleFile));

function handleFile(e) {
    let files = [].slice.call(this.files);
    let largeFiles = files.filter(({size}) => size > MAX_SIZE);
    
    // if there are large files, show a message and prevent upload
    if  (largeFiles.length) {
        let errors = largeFiles.map(({name, size}) => {
            return `${name} is larger than ${MAX_SIZE_STR}`;
        });
        
        errorWrapper.innerText = errors.join('\n');
        submitButton.disabled = true;
        // prevent upload by removing input name (restored with valid input)
        this.dataset.name = this.name;
        this.name = '';
    } else {
        errorWrapper.innerText = '';
        submitButton.removeAttribute('disabled');
        
        // restore input names
        this.name = this.dataset.name;
    }
}

And here's a demo

http://output.jsbin.com/lulopi/

[Mikie]

Hey @Robin S and @abdus thanks for the tips, both look like viable solutions and are good to know!

[a-ok]

On 10/20/2017 at 7:11 AM, Robin S said:

@mke, there is a property and helper method in InputfieldFile that allows a maximum files size to be set but there is no config field for it in admin and you would need to set the max file size in a hook in /site/ready.php:

$wire->addHookBefore('InputfieldFile::render', function(HookEvent $event) {
    /* @var InputfieldFile $inputfield */
    $inputfield = $event->object;
    if($inputfield->hasField == 'YOUR_FIELD_NAME') {
        $inputfield->setMaxFilesize('2m'); // Set the max file size in bytes or use string like "30m", "2g" "500k"
    }
});

Also, for some reason max file size settings of less than 2MB are not enforced - see here.

Do you know if there is anything similar for InputfieldImage? It doesn't look like it even though it's an extension of InputfieldFile.

[dragan]

I would try upgrading to the latest dev version. Ryan mentioned something in his most recent weekly update:

Quote

@BitPoet suggested that our ajax file upload capture and write the uploaded file in ~8 megabyte chunks. This is preferable to loading all the file data into memory and then writing it, enabling it to support larger file uploads that might not have been possible before (depending on server memory). Presumably this also can help to reduce server load. Thanks to BitPoet for writing the code to make it happen.

I don't know if this will also be the case for image-fields though; didn't have time to test it out myself...

[Robin S]

4 hours ago, oma said:

Do you know if there is anything similar for InputfieldImage?

Exactly the same as the hook I posted above, but substitute InputfieldImage for InputfieldFile.

[a-ok]

1 hour ago, Robin S said:

Exactly the same as the hook I posted above, but substitute InputfieldImage for InputfieldFile.

I tried this just as you posted but no such luck... image seemed to upload no problem even though I set 2M and the image was 7M?

[a-ok]

1 hour ago, dragan said:

I would try upgrading to the latest dev version. Ryan mentioned something in his most recent weekly update:

I don't know if this will also be the case for image-fields though; didn't have time to test it out myself...

Maybe a combination of this and @Robin S‘s suggestion?

[Robin S]

5 minutes ago, oma said:

I tried this just as you posted but no such luck... image seemed to upload no problem even though I set 2M and the image was 7M?

Working fine here:

$wire->addHookBefore('InputfieldImage::render', function(HookEvent $event) {
	/* @var InputfieldImage $inputfield */
	$inputfield = $event->object;
	if($inputfield->hasField == 'image') {
		$inputfield->setMaxFilesize('2m'); // Set the max file size in bytes or use string like "30m", "2g" "500k"
	}
});

But if you have client-side resizing enabled for the field that will likely interfere with the checking of the filesize before upload.

[a-ok]

11 hours ago, Robin S said:

But if you have client-side resizing enabled for the field that will likely interfere with the checking of the filesize before upload.

It was this ??‍♂️

Much appreciated help here!

P.S Do you know why we can't set a max-file size lower than 2mb?