Progress on ProcessWire 2.2 and overview of multi-language support

[formmailer]

My first response would be: as much as possible, but with English as a fallback (just like it is now), so if there is no translation available for a module or something else...no problem PW will continue to work as smooth as always.  

But it does mean more work for the translaters. But, maybe even more serious, it might require that PW will use the language files for English as well, no more US-English in the code.  ;D In this way translators can keep the translations up to date with the English files, which wouldn't be possible otherwise.

A large disadvantage might be that the module development becomes more difficult or at least different and it would require very good documentation about how you make your modules multi-lingual. And every module should at least contain English language files.

All of the above would have a major impact on the PW community... but it might be worth it.

/Jasper

[Nico Knoll]

I would split it in two parts: Core translations that will be updated in every new Version of ProcessWire (like

Pages (only the default things not the custom fields), Setup > Fields, Setup > Templates, Setup > Languages, Modules, Access)

and Custom translations which can be added by the module itself if it has the right translation for the selected language.

So everything could be translated but don't have to.

Greets,

Nico

[slkwrm]

I'm also voting for a possibility of full translation with fallback to English, maybe add some guidelines on what are the modules essential for translation, or make something like core translations and optional translations.

I found this question on stackoverflow about cyrillic characters and json, not sure whether it can be helpful, also found on php.net/json_encode that JSON_UNESCAPED_UNICODE as GGTR1138 already noticed is only for PHP 5.4.0, but there are some tips in comments on how to mimic this function in PHP 5.3. So maybe it will help.

@Ryan Yes, I guess Russian is really fancy looking for everybody using latin symbols. I remember many times playng computer games I would run into some weird scriptures on walls of Russian villain's bases like "ЫЗДЦ" or "ЧУЦФАД". The funny thing they didn't mean anything and was there just to make player feel that they are in hostile Russian environment  

[ryan]

Edit : just discovered JSON_UNESCAPED_UNICODE need PHP 5.4.0...

Darn, I thought you'd found it. And I think you have, but looks like we can't take advantage of it just yet. At least not with that particular option.

My first response would be: as much as possible, but with English as a fallback (just like it is now), so if there is no translation available for a module or something else...no problem PW will continue to work as smooth as always.

So everything could be translated but don't have to.

It sounds like most agree that it's best to provide the option for everything to be translatable, but don't require everything to be translated. That's pretty much the way it's setup now, except that I haven't yet made those other modules translatable... I will plan to do it.

But it does mean more work for the translaters. But, maybe even more serious, it might require that PW will use the language files for English as well, no more US-English in the code.  In this way translators can keep the translations up to date with the English files, which wouldn't be possible otherwise.

I want to mention that PW doesn't actually require any language files to run. The language in the code is assumed to be English, and the phrases that appear in the code are the English translation. Those English translations are hashed (md5) and then serve as the tokens for all other translations. So it's reasonably easy for us to make something translatable, even if there aren't any translations available for it.

The English language you see in the Languages list right now is actually just a placeholder for the language already present in the code. But you could make your own English translation (if you wanted to) just by adding files to it.

A large disadvantage might be that the module development becomes more difficult or at least different and it would require very good documentation about how you make your modules multi-lingual. And every module should at least contain English language files.

The good news is that it really doesn't make module development more difficult. The essence of it is that you just have to wrap your text phrases in a function call, so that $value = "text"; becomes: $value = $this->_("text"); The Wordpress i18n documentation will be our starting point for the documentation (assuming it's open source, I think it is) and I'll modify it to be ProcessWire specific and call out any differences.

Once your module is sending its text through $this->_('text'), if you want to add a translation to another language, you just go to Setup > Languages > [any language] > files > new, and then type in the path to the file. I'm guessing that some module developers will choose to include json translation files of their module, and some won't. For those that do, one would just upload that json file to their target language in PW and the given module would translate.

[formmailer]

I want to mention that PW doesn't actually require any language files to run. The language in the code is assumed to be English, and the phrases that appear in the code are the English translation. Those English translations are hashed (md5) and then serve as the tokens for all other translations. So it's reasonably easy for us to make something translatable, even if there aren't any translations available for it.

The English language you see in the Languages list right now is actually just a placeholder for the language already present in the code. But you could make your own English translation (if you wanted to) just by adding files to it.

Would it be possible for translators to see if change in Processwire require changes in the translations?

Let's say you decide that you add something to the ProcessProfile-module, e.g. you add a field for Telephone number. How would a translator know that he/she should update the translation?

If you had a "default English" translation, the translator would see a change there... but since this default translation doesn't exist (for good reasons I think) this trigger doesn't work. So we need something else...

The good news is that it really doesn't make module development more difficult. The essence of it is that you just have to wrap your text phrases in a function call, so that $value = "text"; becomes: $value = $this->_("text"); The Wordpress i18n documentation will be our starting point for the documentation (assuming it's open source, I think it is) and I'll modify it to be ProcessWire specific and call out any differences.

Once your module is sending its text through $this->_('text'), if you want to add a translation to another language, you just go to Setup > Languages > [any language] > files > new, and then type in the path to the file. I'm guessing that some module developers will choose to include json translation files of their module, and some won't. For those that do, one would just upload that json file to their target language in PW and the given module would translate.

That's easy enough. I am glad that it will be so easy.

/Jasper

[ryan]

Would it be possible for translators to see if change in Processwire require changes in the translations?

Whenever you click 'edit' for a translation file, it identifies any translations that are no longer current at the top. It refers to them as 'Abandoned Translations' because the original English translation has changed.

See the attached screenshot for an example of what I mean. It shows that the original field that was previously worded "Hello, how are you doing?", has been changed to "Greetings, how are you doing?". The field has been blanked, and the translation has been moved to become an Abandoned Translation.

We can't make guesses about the scope of the change, so the system has to abandon the translation and let the translator decide what to do with it. The translator may decide that the change is insignificant, and they can just copy/paste their abandoned translation back into the field. Or they may decide to delete the abandoned translation and perform an entirely new translation.

If new phrases are added that weren't previously there, then they just show up as new/blank fields to be translated. I maybe able to call these out in some way more prominently too.

[apeisa]

I think the cleanest solution will be to allow everything to be translated, at least on the long run. But I also think that we should have some documentation of which files are needed for 100% client translation. It might be "newbie" friendly to have everything translated, but if you want to develop using Pw, you do need some basic English (API and Docs are in English) .

I myself have zero interest to translate developer stuff to Finnish. It actually makes developing much harder, since all the examples, docs etc assume English and it is sometimes very hard to know Finnish words for computer/coding jargon (usually there is some clunky official translation used). In Finland most companies code and document in English anyways (no matter if all coders are Finnish).

[slkwrm]

I think the cleanest solution will be to allow everything to be translated, at least on the long run. But I also think that we should have some documentation of which files are needed for 100% client translation.

I'm also voting for a possibility of full translation with fallback to English, maybe add some guidelines on what are the modules essential for translation, or make something like core translations and optional translations.

It looks like we have very similar thoughts on this topic

Edit: Apeisa, I visited Helsinki several days ago, great city, liked it so much, I wish I could stay there longer then just for one day.

[formmailer]

Hi Ryan,

The idea of abandoned/new translations would definitely work (great thinking!!), but, in my opinion, you would need one more thing: an icon (and maybe a text) that triggers the translator.

Maybe something like the image below (and of course with a different icon for new translations  8))

/Jasper

[apeisa]

It looks like we have very similar thoughts on this topic

Spot on! I'm sorry I read all the answers in little bit hurry and from my mobile - It would have been enough to quote you and put +1 there

Edit: Apeisa, I visited Helsinki several days ago, great city, liked it so much, I wish I could stay there longer then just for one day.

Yeah, it is nice city. But I do like St. Peterborough better That is probably my favorite city. 

[ryan]

The idea of abandoned/new translations would definitely work (great thinking!!), but, in my opinion, you would need one more thing: an icon (and maybe a text) that triggers the translator. Maybe something like the image below (and of course with a different icon for new translations

I definitely like the idea, but don't think I could make it scalable from a development standpoint. It's a rather heavyweight process to parse all the files to detect these things. Once we get to having a hundred translation files in there, I don't think everyone would want to wait 20 seconds for it to parse everything every time the language page loads. I may have to take a LazyCron approach with this to let it do the heavy lifting behind the scenes when you aren't looking.

I think the cleanest solution will be to allow everything to be translated, at least on the long run. But I also think that we should have some documentation of which files are needed for 100% client translation. It might be "newbie" friendly to have everything translated, but if you want to develop using Pw, you do need some basic English (API and Docs are in English) .

I will plan to do this. I may not be able to cover 100% of everything (like getModuleInfo title/summary fields), but will find out as I do it.

Yeah, it is nice city. But I do like St. Peterborough better  That is probably my favorite city.

My favorite Finnish city today is Turku I watched a TV show last night about the big ships they build there at the Aker (STX) shipyard, amazing projects!

[apeisa]

OT: This was floating around Twitter and is pretty fun read: http://latimesblogs.latimes.com/jacketcopy/2008/11/why-finnish-is.html (Why Finnish is cooler than English).

[formmailer]

I definitely like the idea, but don't think I could make it scalable from a development standpoint. It's a rather heavyweight process to parse all the files to detect these things. Once we get to having a hundred translation files in there, I don't think everyone would want to wait 20 seconds for it to parse everything every time the language page loads. I may have to take a LazyCron approach with this to let it do the heavy lifting behind the scenes when you aren't looking.

How about not showing the icons by default, but add a "check translations"-button that checks for abandoned and new translations and displays the icons? That means that the heavyweight process only needs to run when you really want it to run.

/Jasper

[Pete]

It might even be simpler if it's just colour-coded. Red for abandoned, green for new? Anything to make it stand out really.

[formmailer]

It might even be simpler if it's just colour-coded. Red for abandoned, green for new? Anything to make it stand out really.

True, but as a colorblind person I should warn for red and green. These colors can be quite confusing for users with colorblindness.

/Jasper

[Nico Knoll]

How about not showing the icons by default, but add a "check translations"-button that checks for abandoned and new translations and displays the icons? That means that the heavyweight process only needs to run when you really want it to run.

I would like to have this, too. Because now I have to open EVERY file to look what's changed. And if I just would have a list with the changes and a quick edit for them it would be a lot easier to have the translations up to date.

[Pete]

True, but as a colorblind person I should warn for red and green. These colors can be quite confusing for users with colorblindness.

/Jasper

Well I mentioned those colours as they're currently used for success/error messages in the system. They can of course be changed via CSS. The main point I was getting at was to try and use CSS that's already part of jQuery UI for simplicity.

[ryan]

I just committed an update that adds counts of abandoned and blank translations with each file (when applicable). You'll see this if you pull in the latest commit, and a screenshot is also attached.

I was worried about the overhead/slowness of parsing all those files on every page load. However, I went ahead and tested it anyway. I was surprised not to see any apparent slowdown – it seems quite fast. It is doing some heavy lifting, but apparently not the scale of issue that I thought. Maybe that will change once we get lots more files in there, but for the moment it seems to be fine. Let me know if you experience something different. I will probably add a caching mechanism if it becomes slow.

[formmailer]

This works great, Ryan. Forget my icons ;D, this is what I/we needed and I immediately spotted one translation I missed in the Dutch language files. (fixed and updated it on Github).

/Jasper

[ryan]

Thanks for testing it out Jasper, glad its come in handy already.

[ryan]

Thanks for the updated French translation!

I pushed an update this morning that changes one aspect with regard to the 'default' language selection. Now you no longer select a default language from the LanguageSupport module config screen. Instead, the default language is the one installed by the module. Previously it was called "English". Now it's just called "Default" and you rename it to whatever you want. If your default language is French, then you rename it to French and upload your French files there. The reason for this is two-fold: 1) Lots of sites don't need English; and 2) The default language is not deleteable. So I figured it's better for those that just need French (for instance) to only have French and not bother with having some non-deleteable English language installed in the system. Besides, since PW is in English when untranslated, any language that you create is already using English until you put in translation files to change it. So there's no reason to have a permanent English placeholder language.

Also, I've added a new module to the system and it's ready for translation. To translate it, click 'new' at the bottom of your translation files and paste this in:

/wire/modules/Process/ProcessPageClone.module

Don't worry about updating your translations just yet because there are still more modules being translated. So you may just want to wait and do them all at once. But I just wanted to mention that one above because it's a new one rather than one that's already in the system.

[Nico Knoll]

Updated german. (http://processwire.com/talk/index.php/topic,598.msg5904.html#msg5904) It now supports cloning

[ryan]

Thanks for the update Nico.

[formmailer]

I just updated the Dutch translation (https://github.com/formmailer/nl-nl-Processwire) and the Swedish translation (https://github.com/formmailer/se-se-Processwire). I added ProcessPageClone to both of them.

//Jasper

P.S. The Dutch translation for "pages" is "pagina's". This works fine everywhere except in the breadcrumbs of the admin:

PAGINA'S > HOME >

Is it possible to fix this small encoding issue?

[ryan]

Thanks for the update Jasper. Looks like we've got a double encoding issue there–thanks for pointing this out, I'll implement a fix. For security, I have everything coming out of the translator as pre-encoded – prevents those JSON files from ever being used for anything malicious. But there are some instances (like that breadcrumb) that may be doing their own encoding (resulting in double encoding). Please let me know if you come across any others.

Thanks,

Ryan