Unable to upload an image with non-ascii (Chinese) characters

[hzesen1221]

I tried uploading an image with an English filename which went through perfectly fine. Then I copied and pasted the same exact file and gave it a Chinese name instead. Next I attemped to upload this image however I got this error message:

"Unable to move uploaded file ..."

I am not sure how to solve this problem. Any help would be greatly appreciated!

[pwired]

Hi there !

The forum of processwire has more than 4 years of posts and has become a great source of information.

Your answer is already in there. Did you have a look ?

Welcome to processwire and why not tell us something more and introduce your self.

[Ivan Gretsky]

This is made by design to avoid problems with handling files on wide range of server OCs. At least, this is how I inderstand it. You can solve the problem by renaming files on upload with this Adrian's module. Have not tried it myself, so please report back if you choose to go this way ))

Actually I would like to read more about this myself, if someone could share some links.

[hzesen1221]

Hello Ivan thanks for your help. I actually tried that module before posting this thread. Unfortunately it did not work for me. For files with English names it worked like a charm. However that is not my problem here. For files with Chinese names I still got the same error. 

And hello Pwired (and anyone else). I am the owner of a small web development firm here in China. However I studied in the U.S. for the last five years so I am exposed a lot to all these 'foreign technologies' and we used these tech exclusively to differentiate ourselves. My company is currently using CouchCMS to develop sites for clients. However I am looking for another more extensible alternative and I found Processwire. I just have to make sure that it will work properly for my clients. 

[hzesen1221]

Update: If the image's filename contains ONLY Chinese characters I got the same error as before. However, if it has any ascii character in it (eg: numbers, letters, punctuation), for example: “未命名-1.jpg”, the module will kick in and rename the uploaded file name for me. Could anyone explain this phenomenon to me? 

Thank you very much.

Zesen 

[Ivan Gretsky]

Just guessing: the module renames only those characters it is aware of. So "未命名-1.jpg" gets renames to something like "____-1.jpg" and if there is no other files whick got renamed to same kind of nonsense name there would be no filename conflict. So you can upload "未命名-2.jpg" to the same field, but not "名未命-1.jpg".

Probably there is a way to include some transliteration table/array for simlified chinese to the module, but Adrian has to step in here (as I may be wrong about all this).

[adrian]

Hi guys,

A quick guess here - I think PW itself is likely stripping all the non latin characters from the filename as the first step. So if there are no latin characters at all, there is nothing left and so the file is rejected. If there is at least one latin character then it can continue to the point where my module hooks in and sets the new filename.

I think these are the two relevant spots in the PW core:

https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/core/WireUpload.php#L213

https://github.com/ryancramerdesign/ProcessWire/blob/a8024bb49785370aa2e3867bd683094663cfeabf/wire/core/Pagefiles.php#L264

Not really sure what can be done about this at the moment. Here is one other relevant post:

https://processwire.com/talk/topic/4387-is-it-possible-to-retain-non-english-filenames/

[Ivan Gretsky]

Please help me on this!

I followed your links, Adrian, and it seems like in validateFilename function there is already a possibility to get the transliterated name for a file by passing "TRUE" instead of "Sanitizer::translate" to filename method.

$value = $this->wire('sanitizer')->filename($value, Sanitizer::translate); 

I am so unexperienced in this, that I may be completely wrong. And if I am not, I still do not know how to hook and change that constant.

But if there is a way to transliterate a name for a page there must be a way to transliterate a filename. We just need to find out how )))

[adrian]

Actually I was just working on filename cleaning for Migrator and learned a little more

I played around with sanitizer->filename and didn't achieve what I wanted, so I ended up making use of:

https://github.com/ryancramerdesign/ProcessWire/blob/7b9730d43e6eda8bd27841a4811a1b9737bfe903/wire/core/Pagefiles.php#L302

public function cleanBasename($basename, $originalize = false, $allowDots = true, $translate = false)

Does that translate option help for you guys?

This is where it is used during Pagefile::install (which is when a file is added to the system)

https://github.com/ryancramerdesign/ProcessWire/blob/7b9730d43e6eda8bd27841a4811a1b9737bfe903/wire/core/Pagefile.php#L91

Notice that it is set to true here, even though the default for the function is false.

[hzesen1221]

Hello Adrian. Thanks for your answer! However I'm still not quite sure what you were suggesting.. Could you please elaborate a little bit more? I looked into that cleanBasename function and sort of understand what it does / what its purpose is. I'm pretty sure you are not suggesting that I should modify this core function right?

Thank you very much.

[adrian]

Sorry - no I am not suggesting you modify the core - it was just meant to be some info to show where the sanitizing of the name is occurring.

I just tested an image upload that I named: 名未命.png and it worked perfectly with Custom Upload Names installed - it was renamed to exactly what the rule specified. Without it installed it still uploaded but got renamed to png.png which is a bit weird, but it still worked.

What version of PW are you running. I am running latest dev. Does it make a difference for you if you upgrade?

[hzesen1221]

I am actually running the latest stable version that I downloaded a few days ago, which I believe is 2.5.2. After a fresh installation I am still getting this error: 

名未命.png - Unable to move uploaded file to: /var/www/..........

Maybe I should try the latest dev version as well however I don't think that's gonna make a difference though...

Do you guys have any idea regarding to why this is happening to share with me? I would really appreciate any inputs!

[adrian]

It seems like it might be server specific. I just tried again on my local dev setup and it works as I reported in my last post. But then I just tried via lightning.pw and it failed with the same message you are getting.

I honestly don't know why though. Do you have another environment you might be able to test with?

[hzesen1221]

I tried another hosting but unfortunately I am getting the same error. I might need to fall back to Perch CMS for this upcoming project then, which I think does less for more (although it's indeed easier than Processwire to set up in my opinion). I will come back and dig a little bit more into this issue in the future. Thanks for all your help! And please keep me posted if any solution is found

[Philipp]

It seems like it might be server specific. I just tried again on my local dev setup and it works as I reported in my last post. But then I just tried via lightning.pw and it failed with the same message you are getting.

Thakns for the Info. We might also have to look into this issue. Maybe our server admin can figure it out and then we can report back.

[hzesen1221]

Thakns for the Info. We might also have to look into this issue. Maybe our server admin can figure it out and then we can report back.

Thanks! Looking forward to it.

[hzesen1221]

Report:

So I ended up modifying the core to cope with this... (yeah I know this is not a good idea but I need this to be functional for my Chinese clients)

In WireUpload.php line 240 I added:

$filename = 'file-' . $filename;

after:

$filename = $this->getTargetFilename($filename);

$filename = $this->validateFilename($filename);

This will force every file uploaded to start with a 'file-' prefix so that the system will not have files with a filename consisting Chinese characters only.

However I ran into another problem:

If I upload 'test.jpg', I get 'file-test.jpg' which is good.

But if I upload '未命名.jpg', I get 'file-' instead, missing the extension.

Then, if I upload '未命名.jpg' again, I get 'file--1.' again missing the extension but with a trailing dot...

This inconsistency is really just weird to me. For English filenames I got the extension, for the first Chinese filename I got the filename without the extension. And if I upload another file of Chinese filename, I got the filename and the dot (e.g.:  file--1. )

Despite all this, I can still view these files as images in my browser, so ...../file-test.jpg , ...../file- ,  ........../file--1. all display the image without any error.

If I insist I need to modify the core in order to make this work (before any official fix is available). Where is the best place to do so? The way I did it works in some sense but produces all these weird filenames which are not really desirable.

Anyone familiar with the core willing to help out here? Thank you so much in advance!

[Ivan Gretsky]

I just played with uploading your files and it worked fine (maybe not so fine, but worked) for me without Adrians module on PW 2.4.14. Got the same results as Adrian. Maybe png.png is not the best option and should be changed, but it does function. I tried moving around those hieroglyphs so I get two 3-symbol files with untranslatable symbols in the filename and it still worked. I got  "png.png" and "png-1.png" or something like that.

Working on Windows 7 with XAMPP.

[Ivan Gretsky]

This issue still exists on (at least my) linux servers. If a file to upload does not contain any latin characters, you get the "Unable to move uploaded file to: ..." error. If a file name has at least any latin character, the upload works. On windows everything works fine. Can we do something about it?

[Ivan Gretsky]

There is continuation of this discussion with some solutions on github. This seems to be the problem with locale settings.