Howto create and add language files to own modules

[cwsoft]

Hello,

after reading forums, old PW weekly and reading through language support docs and Ryans HelloWorld module, I am still struggling to add two translations in German and English to my own module. How would I create the CSV file(s) with description and the hash (md5?). Is there a module used to create the language files, or just create manually? If manually, how is the hash created, or what string/values are used as input to the hash function? 

Any tip, hint or link would be appreciated. I must have overseen some important parts so far. Thanks in advance.

[AndZyk]

Hello @zx80,

I have no experience with translating modules, but have you tried the instruction of the ProcessHello module?

Quote

Bundling multi-language translations with your module

---

This requires ProcessWire 3.0.181+ and that you have multi-language support installed.

• Locate the files you want to translate from your admin: Setup > Languages > language > Site files > Find files to translate. Select the file(s) and submit. ProcessWire will generate new empty .json files for the files you selected to translate.

• In Setup > Languages > language, click the "edit" link for file(s) added for your module. Translate the text into the desired language and save. Near the top of the translation screen is a link to "Download a CSV file". Click that to save the CSV file of translations.

• Copy the CSV file(s) you downloaded in the previous step into a /languages/ directory in your module’s path. For instance /site/modules/ProcessHello/languages/ is the one you'll see with this module. While not required, I recommend naming your files with the ISO-639-1 language code. For instance, German would be de.csv, Spanish would be es.csv, Finnish would be fi.csv, etc.

• If your module has multiple translatable files, you can bundle all the translations into a single CSV file (just copy and paste into one), or you can have multiple .csv files for each language. For instance, if this module had both Helloworld.module and ProcessHelloworld.module files, we might choose to name our csv files es-main.csv and es-process.csv. Or we could just have an es.csv file that merges that translations from both of them.

• In your module’s documentation, instruct the user to install translations from your module’s info/config screen. It’s in the “Module Information” fieldset “Languages” row, where there is an “install translations” link. When new versions of your module also update the translations, make note of that in your changelog so that users will know to click the “install translations” again to update the translations.

Source: https://processwire.com/modules/process-hello/

It seems that you create those CSV files via the backend and then download them to bundle them with your module.

Regards, Andreas

[cwsoft]

@AndZykThanks. Seems to be the missing link. Thanks for pointing me into the right direction. Will try tomorrow and see if I can make it. 

[bernhard]

Adding translations to modules sucks.

That's why I built RockLanguage: https://processwire.com/talk/topic/27199-rocklanguage-a-new-way-to-ship-processwire-modules-with-translation-files-%3F/

The idea is that the module author is responsible for the translations and that it's as simple as possible to translate everything and keep everything up to date.

I have not used it for a while though and it never got any traction in the community, but I still think it is the best way to do it. The official way is by far too tedious. There are so many necessary steps for just translating a single string. And if something changes you have to do everything over and over again. Terrible.

RockLanguage automates all that for you. If you find that anything does not work any more let me know and I'll fix it timely. If you want to help on developing the concept further let me know!

[cwsoft]

@bernhardThanks tor the module. Have read about it on your Github/project site but as usual wanted to try the PW core way first. However I struggled what the core way is or the way it works.

Initially I thought PW supports language files automagically by just putting a key/sprintf per line to translate into a CSV file per language/module. But than I checked out the hello world module and found there is an 'en' and a 'foreign' language column, a description and a hash. Didn‘t realize one needs another module to translate strings, as I never worked with__() functions yet. 

[cwsoft]

Ok, managed to do it the ProcessWire core way by following the steps at the bottom of the Helloworld module by Ryan Cramer. Updated my first PW module for obfuscating emails accordingly.

[bernhard]

Am I the only one thinking that the core way sucks? 

Quote

Bundling multi-language translations with your module

---

This requires ProcessWire 3.0.181+ and that you have multi-language support installed.

• Locate the files you want to translate from your admin: Setup > Languages > language > Site files > Find files to translate. Select the file(s) and submit. ProcessWire will generate new empty .json files for the files you selected to translate.

• In Setup > Languages > language, click the "edit" link for file(s) added for your module. Translate the text into the desired language and save. Near the top of the translation screen is a link to "Download a CSV file". Click that to save the CSV file of translations.

• Copy the CSV file(s) you downloaded in the previous step into a /languages/ directory in your module’s path. For instance /site/modules/Helloworld/languages/ is the one you'll see with this module. While not required, I recommend naming your files with the ISO-639-1 language code. For instance, German would be de.csv, Spanish would be es.csv, Finnish would be fi.csv, etc.

• If your module has multiple translatable files, you can bundle all the translations into a single CSV file (just copy and paste into one), or you can have multiple .csv files for each language. For instance, if this module had both Helloworld.module and ProcessHelloworld.module files, we might choose to name our csv files es-main.csv and es-process.csv. Or we could just have an es.csv file that merges that translations from both of them.

• In your module’s documentation, instruct the user to install translations from your module’s info/config screen. It’s in the “Module Information” fieldset “Languages” row, where there is an “install translations” link. When new versions of your module also update the translations, make note of that in your changelog so that users will know to click the “install translations” again to update the translations.

Stop by the ProcessWire forums anytime and we will be glad to help with any questions.

 

[cwsoft]

@bernhardGuess the core way is fine if you deal with just a couple of translation strings in one or maybe two PHP module files and only support one or two additional languages to the default English (like German in my case) out of the box. But this way may suck if you have 20+ strings spreaded over 5+ files and want to support 3+ languages. 

[teppo]

On 6/17/2023 at 11:55 AM, bernhard said:

Am I the only one thinking that the core way sucks? 

How and why does it suck?

Serious question — I've started bundling translations with my modules, and so far it seems like a nice option. Would be interesting to hear other opinions as well, though ?

[cwsoft]

@teppoSo far I used the core translation for two of my modules. I had only one to four strings to translate into German, as English was the default. I simply don‘t get used to translate the text inputs via backend form input fields per language, then create a CSV file from it and put it in the language folder of my module or template. Biggest hurdle for me seems that those language files are not automatically detected when another user installs my modele. So I need to guide users of my module how to install the supported module languages after installing my module. 

I would prefer just to throw in a CSV file per language into my module language folder without the need of a path/domain or hash value at all. So _t("English string") just uses the stuff from my module CSV file and falls back to English if the translation for the user defined language does not exist. Ideally I just copy the English CSV file, translate it to German and store it as de.csv. No need to translate fields via text inputs form elements or to install languages on top of the module installation. Would find it easier to translate just the CSV file e.g. via VS Code or Notepad++.

P.S. However I am happy to have a core way build in I by default via _t() function instead of writing my own routine or dealing with language based arrays myself. Just would prefer a simpler way of translating my language files via text editor and to not need to install those files already in the module folder manually.

 

 

[psy]

Learnt the hard way that translations were a nightmare, both backend and frontend.

Exuberant client wanted site in 15+ languages. There was a frontend language converter process module that relied on InputfieldCKeditor. Didn't work for me as the new preferred HTML inputfield uses TinyMCE. So I did a thing...

Built my own module to convert all page active multilang fields via Google Cloud Translate. Works well and admin has the choice to enable/disable on page save.  But currently has client-specific limitations and handling things like repeaters requires PW knowledge. It's frontend only.

Happy to collaborate with another PW dev to make it viable for all and add to Download Modules.

Let me know if you're interested.

[bernhard]

@psy I'm not sure what you are talking about ? The topic is about how to create and add translations to modules. If I understand you correctly you complain about the whole translation system in PW? My statement was solely about the process of getting translations into the modules folder ? 

I don't even have a problem with the translation via the PW backend, though I see that it could be nice to have a CSV file that you can send to a client and get back a translated file that you can then back import into the sytem. We have had that request once but then the client didn't want it. The problem with translating stuff in the backend is that you need to provide a login for that and need to make sure that all permissions are setup correctly and even then I heard from clients that this is "too complicated / I'm afraid, why does it not work like WordPress there I know how to do it".

@teppo I already wrote what I don't like about the process. Imagine you have a module and you add one translatable string to that module. The steps necessary to push that translation into your modules folder are:

Quote

Long version:

Locate the files you want to translate from your admin: Setup > Languages > language > Site files > Find files to translate. Select the file(s) and submit. ProcessWire will generate new empty .json files for the files you selected to translate.

If you selected just one file above, it will take you there now. Otherwise, in Setup > Languages > language, click the "edit" link for file(s) added for your module. Once on the file translation screen, translate the text into the desired language and save (I just copied/pasted from Google Translate for the two modules above). Near the top of the translation screen is a link to "Download" or "View" a CSV file of the translations. Click that to save (or view) the CSV file of translations.

Copy the CSV file(s) or translations you exported in the previous step into a /languages/ directory in your module’s path. For instance /site/modules/ProcessHello/languages/ is the one you'll see with the ProcessHello module. While not required, you might consider naming your files with the ISO-639-1 language code. For instance, German would be de.csv, Spanish would be es.csv, Finnish would be fi.csv, etc. Regardless of what name you use, the goal is just simply that the filename should explain what language it is for.

Since a single .csv file can provide translations for multiple .php files, if your module happens to have multiple translatable files, you can bundle all the translations into a single CSV file (just copy and paste into one), or if need be, you can have multiple .csv files for each language. Though I prefer to keep it simple with one file per language.

In your module’s documentation, instruct the user to install translations from your module’s info/config screen. It’s in the “Module Information” fieldset “Languages” row, where there is an “install translations” link. When new versions of your module also update the translations, make note of that in your changelog so that users will know to click the “install translations” again to update the translations.

Steps necessary when using RockLanguage and when RockLanguage is setup for your project (you have to do that only once per project):

• Once on the file translation screen, translate the text into the desired language and save
• git commit

Now imagine you pushed your translation and then you add another word the other day... What would you have to do?

Quote

Long version:

Locate the files you want to translate from your admin: Setup > Languages > language > Site files > Find files to translate. Select the file(s) and submit. ProcessWire will generate new empty .json files for the files you selected to translate.

If you selected just one file above, it will take you there now. Otherwise, in Setup > Languages > language, click the "edit" link for file(s) added for your module. Once on the file translation screen, translate the text into the desired language and save (I just copied/pasted from Google Translate for the two modules above). Near the top of the translation screen is a link to "Download" or "View" a CSV file of the translations. Click that to save (or view) the CSV file of translations.

Copy the CSV file(s) or translations you exported in the previous step into a /languages/ directory in your module’s path. For instance /site/modules/ProcessHello/languages/ is the one you'll see with the ProcessHello module. While not required, you might consider naming your files with the ISO-639-1 language code. For instance, German would be de.csv, Spanish would be es.csv, Finnish would be fi.csv, etc. Regardless of what name you use, the goal is just simply that the filename should explain what language it is for.

Since a single .csv file can provide translations for multiple .php files, if your module happens to have multiple translatable files, you can bundle all the translations into a single CSV file (just copy and paste into one), or if need be, you can have multiple .csv files for each language. Though I prefer to keep it simple with one file per language.

In your module’s documentation, instruct the user to install translations from your module’s info/config screen. It’s in the “Module Information” fieldset “Languages” row, where there is an “install translations” link. When new versions of your module also update the translations, make note of that in your changelog so that users will know to click the “install translations” again to update the translations.

I intentionally did not use the forums spoiler feature and I intentionally did not use "same as above" to indicate how much (unnecessary and repeated) work the current process puts on the developer.

Imagine when using RockLanguage:

• translate the word
• git commit

[psy]

@bernhard 

18 hours ago, bernhard said:

I'm not sure what you are talking about ? The topic is about how to create and add translations to modules. If I understand you correctly you complain about the whole translation system in PW? My statement was solely about the process of getting translations into the modules folder ? 

Yes, I was bit off topic. ?‍♀️ Apologies. And yes, I did find the whole translation system confusing and/or tedious. Haven't had to translate modules but when I do, I'll use your RockLanguage module ? 

[bernhard]

1 hour ago, psy said:

Yes, I was bit off topic. ?‍♀️ Apologies. And yes, I did find the whole translation system confusing and/or tedious.

No problem, just tried to make sure we are talking about the same thing ? What do you find confusing/tedious? Maybe it's similar underlying things?

1 hour ago, psy said:

Haven't had to translate modules but when I do, I'll use your RockLanguage module ? 

If you do I'm happy to hear how it goes. I've not been using it for a long time as nobody ever showed some interest and for myself I've just gone the quick route and translate it in place. But that will hopefully change as soon as I release my new website ? 

[teppo]

On 6/30/2023 at 5:05 PM, bernhard said:

 

@teppo I already wrote what I don't like about the process. Imagine you have a module and you add one translatable string to that module. The steps necessary to push that translation into your modules folder are:

Steps necessary when using RockLanguage and when RockLanguage is setup for your project (you have to do that only once per project):

• Once on the file translation screen, translate the text into the desired language and save
• git commit

Now imagine you pushed your translation and then you add another word the other day... What would you have to do?

I intentionally did not use the forums spoiler feature and I intentionally did not use "same as above" to indicate how much (unnecessary and repeated) work the current process puts on the developer.

Imagine when using RockLanguage:

• translate the word
• git commit

Thanks for elaborating! When you put it like that, it definitely seems like a lot ?

In my case (using the core approach) the process boils down to this:

1. Translate the word in admin
2. Click the "view" link to display current CSV content
3. Manually copy and paste the new string from the CSV content in admin to existing CSV file bundled with the module
4. git add + commit

Manual step is required to copy the data, and another manual step is required to update translations for a module on another site, but to me personally it doesn't feel like a very big deal. Additionally I would never allow PHP to write anything directly into the modules directory — in my opinion it is a very risky thing to do, so that option is out of the question ?

Anyway, since the core has a system for bundling translations with modules, it would make sense to see how it could be improved so that perhaps we don't need "competing" solutions. From this thread alone I've picked a couple of ideas that would (in my opinion) make sense as core additions, and I'm planning to open requests or submit PRs for those.

[bernhard]

17 hours ago, teppo said:

Manual step is required to copy the data, and another manual step is required to update translations for a module on another site

Which means 100 more manual steps on 100 other sites, or if there are 3 steps for each site that's 300 manual steps. The example is a little constructed of course, but I just hate these kind of tasks and I'm used to "git pull and it works"

17 hours ago, teppo said:

Anyway, since the core has a system for bundling translations with modules, it would make sense to see how it could be improved so that perhaps we don't need "competing" solutions. From this thread alone I've picked a couple of ideas that would (in my opinion) make sense as core additions, and I'm planning to open requests or submit PRs for those.

That would be great.

17 hours ago, teppo said:

Additionally I would never allow PHP to write anything directly into the modules directory — in my opinion it is a very risky thing to do, so that option is out of the question ?

Could you please explain why that is so risky or what the risk exactly is?

[teppo]

3 hours ago, bernhard said:

Which means 100 more manual steps on 100 other sites, or if there are 3 steps for each site that's 300 manual steps. The example is a little constructed of course, but I just hate these kind of tasks and I'm used to "git pull and it works"

Fair ?

3 hours ago, bernhard said:

Could you please explain why that is so risky or what the risk exactly is?

If PHP has write access to directories containing executable code, this opens a huge can of worms: a malicious or vulnerable module could allow attackers to write code and wreak serious havoc on the target system. Same thing could happen if there's a flaw in site code that allows attacker to write or download their own executable files. You could argue that a malicious or vulnerable module could cause similar problems anyway, but in my experience it is often easier to slip in code that writes code than code that does evil things.

At the very least it's another attack vector.

I'm aware that even the installer suggests that we allow PHP to write to the modules directory, but in my opinion that should not be done unless absolutely necessary. And I have yet to come across a situation where that would be the case ?

[cwsoft]

@teppo With great power comes great responsibility :-). 

So far I got the impression that PW is a great tool for devs to create websites of any kind with an excellent API. As even template files are PHP files by default any non proper usage of PHP commands in templates, modules or hooks can cause serious entry points for an attacker. Guess most critical vectors of attack in 2023 are still not proper sanitized user input used in queries - even PW queries - or inputs output on screen without proper sanitizing in first place (e.g. CSRF attacks, reading JS Cookies or directory traversal).

So yes, eliminating attack vectors is always a good habit devs should care about.