Max file size for uploading a file?

[tinacious]

Hello.

I have a client who wants the ability to upload a large file (~350 MB) and share it.

I'm wondering if there will be any issues with uploading large files through ProcessWire and what should be done to prepare to make sure this all runs smoothly before I begin development.

Thank you for your help.

[Exceptionz Project]

Regardless of whether handling this with processwire you should change your server variables to handle such a request.

Your php.ini should be adjusted for it (at least):

upload_max_filesize = 350M
post_max_size = 350M

And your apache should be configured too:

memory_limit = 400M

Update (thx Wanze)

You should also adjust settings like

max_execution_time
max_input_time

in your php.ini...

I don't know if your clients host has such capabilities (-> memory limitations for such one vhost) but you should consider to convince your client to upload such files with ftp (not with http) and it is less error-prone for you and him with ftp...

[Wanze]

In your php.ini, you should also set the values for "max_execution_time" and "max_input_time" to more than 30 seconds (30 = default value I guess).

But I also suggest to upload huge files via FTP.

[tinacious]

Thanks, that's very helpful!

[nalply]

An additional note to whom it may concern:

For the people who are running ProcessWire under nginx and fpm (like me) there is an additional setting for nginx:

  client_max_body_size 500m;

(see http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size for details).

It also helps to look for error logs.

[bernhard]

hi!

ok i'm now also having problems with file uploads between 500 - 1000MB on plesk.

first problem was an error after some time due to MaxRequestLen on php FCGI

that seemed to fix it: https://www.opensource-excellence.com/blog/item/430-cant-upload-files-due-to-fcgi-maxrequestlen-restriction-in-plesk.html

now the upload proceeds until 100% but throws this error in the console:

InputfieldFile.min.js?v=124-1463995380:1 POST https://.../admin/page/edit/?id=xxx&InputfieldFileAjax=1 net::ERR_EMPTY_RESPONSEn @ InputfieldFile.min.js?v=124-1463995380:1w @ InputfieldFile.min.js?v=124-1463995380:1(anonymous function) @ InputfieldFile.min.js?v=124-1463995380:1

The file is around 700MB. After upload i have a 502MB file in the folder. When i save the page with the stuck 100% upload this file is removed completely (thats the part where i know why ^^ )

Any help for this? I have no errors in the apache error log for the second problem. The first Problem showed this (for reference and search engine):

[Tue May 31 15:34:02 2016] [warn] [client xxx] mod_fcgid: HTTP request length 134225920 (so far) exceeds MaxRequestLen (134217728), referer: https://xxx/admin/page/edit/?id=xxx

[Doug G]

 Maybe take a look at the php setting post_max_size.  Just a wild guess.

[bernhard]

Thank you doug, of course i set all the limits to proper values (like stated in this post).

Any other ideas?

[Doug G]

Well, if you want more wild guesses, do you have any filesystem quotas on your server?

[cstevensjr]

Maybe I am looking at this the wrong way, however when I look at the image (screenshot) you posted, I say to myself "Ok, That shouldn't work"

You have a post_max_size that is the exact same size (800M) as your upload_max_filesize setting.  Even though the explanation below it explicitly states "Typically, should be larger than upload_max_filesize and smaller than memory_limit."

It may not make a difference, but I would have at least set the upload_max_filesize to 750M.  That would actually follow the PHP guidance in the notes on the panel.  In other words, having something the same size is not the same as it being larger (at least where I come from).

Also, if I am dealing with files that are that large (700M), I would be using SFTP or SSH to upload the files to the server.  To me, those are gigantic files. 

You may have everything setup correctly, however the configuration of the server (Plesk/Web Host/Security Settings) may not allow you to post a file this large.  You would need to consult with the hosting provider (if online)

Edited June 2, 2016 by cstevensjr
Added additional notes

[pwired]

No matter what you are going to setup in your php.ini, with uploading files of that size you are probably going to run into your hoster limitations, internet connection time outs, file corruption, etc.

First you should try to embed such files through dropbox, box, gdrive, etc.

If you want to keep everything in the house you can use html5 FileApi,

http://www.html5rocks.com/en/tutorials/file/dndfiles/

or this one: https://github.com/mailru/FileAPI

[LostKobrakai]

The image field is already using the file-api (if supported). But I'd imagine that the reason for pages like wetransfer to use flash instead of native technology.

[bernhard]

Thank you all for your answers.

@cstephensjr different values make no difference... i'll try to contact my vserver hoster

[bernhard]

hmmm...

Quote

wir konnten das Problem nachvollziehen, es scheint hier ein Problem mit Ihrer Anwendung zu geben.
Bitte beachten Sie jedoch das wir keinen Support für Drittanbieter-Software und Kunden-Skripte anbieten.
Leider können wir Ihnen bei diesem Problem nicht helfen. Wenden Sie sich daher bitte an den Hersteller der Software, oder Ihren Entwickler der die Anwendung geschrieben hat.

Mit freundlichen Grüßen

i did not get any more information about WHAT the problem could be... they do not provide 3rd party software support. strange that i am the only one having this problem if it is really related to processwire...

i'll try another upload tool and see what happens.

[LostKobrakai]

I always hate those kind of answers. I mean they invested time to replicate or otherwise prove that it's not their fault, but giving any detailed information about the results cannot be done?! 

[horst]

At least it let enough room for speculations.

• Did they have investigated?
• Is it a server config that they don't want tell about or change it?
• Or do they NOT have investigated and this is a boilerplate answer that is sent with enough timedelay?

annoying!

[bernhard]

at least they changed my php-settings and didn't say a word or changed them back... access logs seem that they logged in but i'm not sure if the REALLY tried to upload a file...

[bernhard]

ok i got another step further: the processwire error logs say the following:

/admin/page/edit/	Error: Allowed memory size of 1048576000 bytes exhausted (tried to allocate 444071936 bytes) (line 144 of /var/www/vhosts/.../httpdocs/wire/core/WireUpload.php)

https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/core/WireUpload.php#L144

what does that mean?? allowed = 1GB, exhausted = 444MB

does anybody have an idea on this?  i'm really getting frustrated...

[bernhard]

hmmmm... update: on a fresh install WITHOUT https it works for a 614MB file.

i switched off HTTPS on the other install, but still it does NOT work. debugging is really painful, as every upload needs >10mins

any ideas what the problem could be or how i can speed up debugging?

edit: got this now in the apache error log when trying to upload a file without https on the client site (only this time!)

[Tue Jul 05 21:45:55 2016] [warn] [client xx.xx.xx.xx] (70008)Partial results are valid but processing is incomplete: mod_fcgid: can't get data from http client, referer: http://.../admin/page/edit/?id=6114

[flydev]

10 hours ago, bernhard said:

got this now in the apache error log when trying to upload a file without https on the client site (only this time!)

What are the values of :

• IdleTimeout
• ProcessLifeTime
• FcgidMaxRequestLen
• FcgidIOTimeout

?

[bernhard]

thank you @flydev i changed my server so i don't have this problem any more...

but i still have a problem i need to solve, but that's another topic. on my research i found this answer: http://stackoverflow.com/a/5828654/6370411

it states that this should work:

 upload_max_filesize = 1G
 post_max_size = 1G
 memory_limit = 32M

that's strange because i'm getting out of memory errors now on uploads via processwire file field...

i will investigate further and hold this here as reference. or does anybody know why memory_limit should not be a problem and IS a problem for processwire?

[LostKobrakai]

I'm not sure what exactly you're building there, but maybe you could use something like flysystem's dropbox adapter to let people upload such big files to dropbox, which is far less error prone, and import from there. Tech. php can handle those files, but some hick-up in your connection (on mobile) or anything will corrupt any upload and you need to start over again.

Edit: Also this is probably not a ProcessWire specific issue, until the file is uploaded 100% it's not even touched by processwire. 

[clsource]

May be using this

https://transfer.sh

could help too.

[bernhard]

thank you all.

for the time i ended up using dropbox and just share the download-links:

[itsberni]

Hello,

unfortunately I have to rehash this thread;-)

I have a field of type "file" and I just can't get files larger than 128M uploaded.
The php.ini settings are as follows:

upload_max_filesize: = 256
post_max_size = 256
memory_limit = 512
max_execution_time = local value: 120 / master value: 300
max_input_time = 600

Is there somewhere in the file settings a hidden possibility to limit the upload volume?

Thank you very much for your input!