Multi-language behavior in modules (special case when using external json data files)

[Gadgetto]

Hi there,

for my SnipWire module I'd like to implement a countries library which provides Json arrays with world-countries and their associated alpha-2 and numeric codes as defined by the ISO 3166 standard.

The library provides the word-countires lists in different languages. I created files like

world.en.json:

[{"id":4,"name":"Afghanistan","alpha2":"af","alpha3":"afg"},
{"id":818,"name":"Egypt","alpha2":"eg","alpha3":"egy"},
...

world.de.json:

[{"id":4,"name":"Afghanistan","alpha2":"af","alpha3":"afg"},
{"id":818,"name":"Ägypten","alpha2":"eg","alpha3":"egy"},
...

and so on ...

A special class handles the loading of the language files and converts them to PHP arrays. The problem is, I don't know how to load the language file based on the current admin language. ProcessWire doesn't use ISO language codes to identify languages. In a default installation the base language is defined as "default". Other languages can be defined as one likes (e.g. "german", "spanish", ...). It is even possible that admins changes the "default" language to German for example.

As this method doesn't use the ProcessWire i18n API ($this->_('Text') or __('Text')), what would be the best and stable way to load the json files based on the correct language? Is this even possible or should I choose another method using translatable strings instead?

-- Martin

[dotnetic]

Hi Martin, in one project I use https://github.com/umpirsky/country-list which provides list of countries and their ISO codes in many formats like JSON, PHP array, CSV, etc.

It is installable via composer.

Cheers
Jens 

[dragan]

It depends where and in how many places you will use the countries list.

You could make it part of the module's config routine: Let the admin map each PW-language to the country-language-set (similar to what we all know when we import CSVs). 

If you need the country strings in more than just one part of the site, you could create a dedicated tpl, page-tree and with multilang title field populate the site's language fields.

[wbmnfktr]

What about...

You have (almost) each and every possible variant in your module/setup available... why don't you let users decide which to install and which set is mapped to whatever language available.

You could list all available/defined languages of a ProcessWire instance and the user/admin could/has to decide which language belongs to whatever set. Similar to the field mapping in FormBuilder (save to pages).

For example:

• default is EN, while
• Deutsch is DE and
• Italiano is IT

Would that work? Would that be enough? 

I don't know. Just an idea.

 

[Gadgetto]

16 hours ago, Jens Martsch - dotnetic said:

Hi Martin, in one project I use https://github.com/umpirsky/country-list which provides list of countries and their ISO codes in many formats like JSON, PHP array, CSV, etc.

It is installable via composer.

Cheers
Jens 

Hey Jens,

thanks for the hint, but the problem isn't finding a library - the problem is, I don't know a good way to load the correct country list file based on the current PW language.

[LostKobrakai]

Languages in PW are just pages. So you can always add fields to e.g. store iso codes in. There's nothing in processwire core, which would be helpful in identifying languages otherwise, as they just use user provided titles (and default). 

[Gadgetto]

@dragan & @wbmnfktr,

your suggested method (the admin needs to map the languages) currently seems to be the way to do this. I just don't think it's very elegant.
After all, I'll probably implement the country list as a standard array with PW language methods. This is the way how PW handles it. I could try to generate the PHP language files with their language methods programmatically.

[Gadgetto]

5 minutes ago, LostKobrakai said:

Languages in PW are just pages. So you can always add fields to e.g. store iso codes in. There's nothing in processwire core, which would be helpful in identifying languages otherwise, as they just use user provided titles (and default). 

Yes, I do this in frontend each time I'm using multi-languages. The problem is, for a ProcessModule I have no warranty that the site owner uses "default" for English language. So a manual mapping would always be required.

[LostKobrakai]

6 minutes ago, Gadgetto said:

So a manual mapping would always be required.

It is. You cannot be sure if default is english or not, but even less for the rest, where you don't know anything in advance about the name. What your module can do though is adding the required field into the template for languages.

Edit: 

More related on the general idea of multi language modules and possibly less your usecase: 

https://github.com/processwire/processwire/pull/52

This PR would allow modules to ship with translation files, which any user can then manually import/map into their language setup.

[Gadgetto]

I think it would be great if the ProcessWire language template already had a field e.g. "alpha-2" which needs to be filled (required) with the language code defined by the ISO 639 standard. Each language could have it's own custom name "default", "german", "english", ... (like it is now) but a language code would be also needed.

I know I can add this field manually, but I think it should be in the PW core - with the restriction that you would have to map this manually!

I think I'll post a feature request for Ryan ...

 @LostKobrakai Your PR from above would be a great addition! I gave your request a "thumbs up"!

[Gadgetto]

@LostKobrakai your post from above made me curious: is there currently no way to provide/preinstall language files with a module? I'd like to release SnipWire at least in English and German. 

[dragan]

1 hour ago, Gadgetto said:

is there currently no way to provide/preinstall language files with a module?

Not as far as I know. (only with site-profiles, but that's a completely different scenario)

I guess one reason is that you probably don't want to install another language, just because a module author has a pre-defined set of languages that don't match with your own site setup.

Though I assume if you would create new templates with your module, and use the __() syntax, PW will automatically create the necessary JSON translation files. Or at least, PW will list them in the language menu (where you can select templates that don't yet have a ML JSON file, and then you can create new json files and fill in the strings on the following step).

<offtopic>
multi-language handling in any system is complex. The more you have to deal with it, and dig deeper into the subject, the harder it gets.
</offtopic>

[horst]

@Gadgetto You may ask the maintainer(s) of the (german) language pack to include it. 

[Gadgetto]

1 hour ago, dragan said:

Not as far as I know. (only with site-profiles, but that's a completely different scenario)

I guess one reason is that you probably don't want to install another language, just because a module author has a pre-defined set of languages that don't match with your own site setup.

Though I assume if you would create new templates with your module, and use the __() syntax, PW will automatically create the necessary JSON translation files. Or at least, PW will list them in the language menu (where you can select templates that don't yet have a ML JSON file, and then you can create new json files and fill in the strings on the following step).

<offtopic>
multi-language handling in any system is complex. The more you have to deal with it, and dig deeper into the subject, the harder it gets.
</offtopic>

Thats really bad (at least in my opinion). In MODX where I came from, this was never a problem. Pre-defined language files could be easily installed withe each plugin. Translations could be done using web-services like Crowdin.

So I can only release my module with a single language. And translation has to be done manually after that within PW.

[dragan]

18 minutes ago, Gadgetto said:

Thats really bad (at least in my opinion). In MODX

That's "where I come from" as well :-)

20 minutes ago, Gadgetto said:

And translation has to be done manually after that within PW.

That has been bothering me for quite some time too - in the context of PW Pro modules that were going to be used in a ML setup.

I had to translate every single string of Lister Pro, Pro Fields.... modules etc. myself, so that the client's backend-admins / authors had the interface available in their own language as well. This - to me - is a massive oversight by Ryan. You buy a Pro module, and probably every other aspect in the work-flow is ML-capable, but the actual backend GUI is not.

[gebeer]

As far as I understand it, you could bundle language translation json files with the module, let's say in a module folder 'translations'. They won't be installed automagically, though. But in the language edit interface, you can add these json files manually to the 'Site Translation files'. Or am I wrong here?

[horst]

4 hours ago, gebeer said:

add these json files manually to the 'Site Translation files'. Or am I wrong here?

That is correct. Additionally you can send the language files to the maintainers of language packs. 

I think that editors don‘t get the core backend untranslated, but need a single module translated. (What also can be done with one additional step, if the language files are distributed together with the module)

For me it doesn‘t look that bad as described by @dragan. 

[Gadgetto]

2 hours ago, horst said:

That is correct. Additionally you can send the language files to the maintainers of language packs. 

I think that editors don‘t get the core backend untranslated, but need a single module translated. (What also can be done with one additional step, if the language files are distributed together with the module)

For me it doesn‘t look that bad as described by @dragan. 

Could you please describe the workflow how I can create those language-files to be later added manually to the 'Site Translation files'?

Edit:

OK, I think I get it partly.

1. I need to translate all SnipCart related files in PW backend and the export as .zip. The .zip holds all translated Json files of the complete sites folder.
2. now I could pick only those Json files which belongs to the SnipCart package and bundle them with the module.
3. an admin has then to import those SnipCart related Json files in PW language editor via this:

If this is the correct way, then its really not as big problem as it seems to.

-- Martin

[gebeer]

24 minutes ago, Gadgetto said:

Could you please describe the workflow how I can create those language-files to be later added manually to the 'Site Translation files'?

Once you want to translate your module, go to Setup->Languages->yourlanguage, choose to translate your module files, do the translation in the interface and then download the resulting json. That should be it.

[dotnetic]

21 hours ago, Gadgetto said:

I don't know a good way to load the correct country list file based on the current PW language.

I read your first post again. So you want to load the correct country list file based on the current PW language in the admin, right?

Well, this is really easy:

Setup your languages with the two letter language code as the name, and as the title enter the name of the language in spoken words:

Then in your module's init function enter this to get the correct file and store it in an array (example based on umpirsky/country-list):

$this->lang = $this->user->language->name;
if ($this->lang == "default") $this->lang = "de";

// then you can load the file from your country list into an array
$this->laender = require wire()->config->paths->root . "vendor/umpirsky/country-list/data/$lang/country.php";

 

16 hours ago, Gadgetto said:

is there currently no way to provide/preinstall language files with a module

No there isn't a way. This was filed as an issue before, and I would also like to see this functionality in the future.

2 hours ago, horst said:

That is correct. Additionally you can send the language files to the maintainers of language packs. 

 

I am the maintainer of the german language file, and would not recommend to do it this way. Because a language pack should not contain translations for a custom module. Else a maintainer would have to update/merge the files continously if a module author changes anything.

The best way would be if PW natively had support for module language files. I think we should talk to @ryan about this, and propose a request in the processwire-request repo.

[gebeer]

3 minutes ago, Jens Martsch - dotnetic said:

Setup your languages with the two letter language code as the name

Problem is, that you cannot predict which names people use for languages. So you cannot rely on a specific standard here. If someone installs your module who has named the German language 'german' and not 'de_DE', the module author has no way to reliably map translation files to country names.

[Gadgetto]

9 minutes ago, Jens Martsch - dotnetic said:

Well, this is really easy:

Setup your languages with the two letter language code as the name, and as the title enter the name of the language in spoken words:

This won't work, as you you have no warrant, that the admin has set "default" to German (or English).

I think the best way (as we discussed above) is, to provide the translated json language files with the module and the site admin has to import those files into the desired language)

Edit: @gebeer you beet me by 2 minutes ...

[dotnetic]

Yes, you guys are right. But the "best way" would be if PW natively had support for this, regardless of what the user has chosen as the language title. The language name should be the defined value from the ISO 3166 standard, although one should be able to change the URL of the language, so instead of "de" the URL "deutsch" could be used.

EDIT: This would enable us module devs to deliver language files with our modules, regardless of the users language setup.

[dotnetic]

I am thinking of something like this:

language name: predefined by ISO 3166
language title: user can chooose
language url: user can choose, fallback to ISO 3166

[Gadgetto]

8 minutes ago, Jens Martsch - dotnetic said:

Yes, you guys are right. But the "best way" would be if PW natively had support for this, regardless of what the user has chosen as the language title. The language name should be the defined value from the ISO 3166 standard, although one should be able to change the URL of the language, so instead of "de" the URL "deutsch" could be used.

EDIT: This would enable us module devs to deliver language files with our modules, regardless of the users language setup.

Thats exactly what I posted above: