Workflow in maintaining translations

[yellowled]

I'd like to share and discuss some things I've come across maintaining the German translation.

1. Ryan has outlined a way to get the translatable files in PW: http://processwire.c...__20#entry11232 — while this does work, it's not very comfortable. It would make translations much easier if there was a way to get those in the backend, as well as maybe detect which translatable files are new or abandoned. For instance, I think InputfieldRadiobuttons just got moved to a separate directory, making the previous translation file uneditable.
   
2. I think it would also be good to have different "states" for empty fields in translations. As of now, "empty" could mean "not translated yet" as well as "doesn't require translation, can fall back to English value". Updating a translation would be easier if one could just skip the values which don't need to be translated, but obviously, this would be nice to have. It's definitely not a must-have.
   
3. Which brings up the question as to when to actually update a translation. Obviously, doing that constantly as changes are committed to GitHub is insane. So it would make sense to update translations based on a release. As far as I have followed PW development, there is no alpha/beta stage, right? It would be nice to have some phase before a release where nothing new is added so translations could be updated to be ready along with a new release.
   
4. Another thought: Let's say I install a translation for PW 2.3 in a PW 2.2 installation (or vice versa). Are there going to be any "side effects"? Should we maintain versions of the translations for specific PW releases?
   
5. Most pressing issue in my humble opinion: currently (as far as I can see, please correct me if I'm wrong), there's no "protection" for additional translations added. Let's say I install a translation, then I add some translations of my own, maybe related to the templates of the site or some help texts in the descriptions of fields. Then I update PW and the updated installation. In that case, all my additional translations will be overwritten by the updated translation, right? If so, there should at least be a way to backup the additional translations, although I figure this could be hard to implement given the JSON format of the translations.
   

Oh boy. Sorry for rambling here, I hope this makes some sense.

[slkwrm]

Yellowled, so many ideas! Nice)

1. I agree that having a tool to detect new translatable files would be great. I tried to write a module that would do this, but then I had some more important things to do and postponed this affair.

2. These are nice features, especially, to have a placeholder for fields that don't need translation and fallback to default. (English, I guess). But are not must have as you said.

3. I can't agree here. It's nice to have a free schedule of translation updates. It's more convenient for community as everybody make their contribution when they have some vacant time. Would be nice to have an ability to download and install translations along with modules via modules repo. Especially if it is possible on the first step of installation, so that even the remaining steps of installation are localized.

4. I don't think it is a viable scenario. PW releases are very stable, so it's not a problem to update to a newer version. For now I would maintain translatoin only for the latest version, maybe it will change in future when Ryan will deside to drop backward compatibility for a new version of PW.

5. All translations are in the translation page /assets/files/ folder (the same as page id), your translations are safe during core updates, but if you update translation files then yes, you can easily erase your changes. What about having an opportunity to specify fallback translation for a hmmm... translation? So if you have your personal, say, German translation files you can fallback to general German which you download from GitHub, so your translation remains even if you update general German files.

Sorry for mistakes. I'm really sleepy now

[yellowled]

1. Being able to export changes would also be nice. No more having to look for that one folder in assets/files.
   
2. Yes, English would be the fallback. Some technical terms are pretty universal or at least acceptable in other languages.
   
3. You're probably right, the "contribute when you have the time" factor is crucial for most open source projects.
   
4. You're probably right. Moreover, maintaining multiple versions is pretty insane as well. Then again, using a GitHub repo, it's not that much work.
   
5. I think it would be good to indicate and/or separate "system" and "user" translations, I'm just not sure it makes sense from a technical point of view.
   

[Soma]

Thanks for sharing your thoughts. I also agree with most being said.

It's nice the way it works and all, but maintaining language packs and template or custom module translation is very hard as it is now.

Some more features that would help for system translations would be greatly welcome. Looking at it now it is very hard and time consuming finding out what's added or what changed. There's some note now how many are left out or abandoned. But that's not enough I guess. Having 73+ translation files adds to it.

Also what annoys me is that to that whole lot of translations comes the custom modules and template translation which are separate and adds even more. So separate them in some way would maybe help already.

Some system comes to mind that maybe there could be some system translation index that will get updated by Ryan on every new realease that could serve as a base finding differences and additions, so it would be easier to keep language packs up to date easily.

Looking at the current german pack there's already some new, abandoned (changed paths) and so and it's very hard to find out what has changes by hand.

[ryan]

Ryan has outlined a way to get the translatable files in PW: http://processwire.c...__20#entry11232 — while this does work, it's not very comfortable.

I could maintain a default translation pack on GitHub, which already has all the files in it (empty, ready for edits). That way, someone could just upload that to start a new language translation, or easily see what's been added (or more rarely, moved or deleted) via the commit history. This would only be necessary for the files themselves, as the LanguageSupport modules are already telling you of changes to the translations. But I'll be happy to maintain this if you guys would find it useful.

It would make translations much easier if there was a way to get those in the backend, as well as maybe detect which translatable files are new or abandoned. For instance, I think InputfieldRadiobuttons just got moved to a separate directory, making the previous translation file uneditable.

This is a rare occurrence in the core. In this particular instance, we needed to add a CSS file to InputfieldRadios. That module wasn't in it's own directory before, so it had to be moved into one. The translation file would probably have to be renamed to reflect the updated location. While abandoned translation files should be rare in the core, a detection for it may still be worthwhile, especially for 3rd party modules since you wouldn't need to keep around translation files for a module that was no longer installed.

Looking at the current german pack there's already some new, abandoned (changed paths) and so and it's very hard to find out what has changes by hand.

There have only been two translatable files in the core moved in the last year that I can think of: InputfieldPage.module and InputfieldRadios.module. Again, this should be a rare occurrence. But another good reason for me to keep a master translation on GitHub, so there is a place where a translator could more easily keep track of it.

I think it would also be good to have different "states" for empty fields in translations. As of now, "empty" could mean "not translated yet" as well as "doesn't require translation, can fall back to English value". Updating a translation would be easier if one could just skip the values which don't need to be translated, but obviously, this would be nice to have. It's definitely not a must-have.

This one may be easy for us to accomplish if we just say that a translation with nothing but a period "." in it might be considered intentionally empty, or something like that. Then I can add a bit of code to detect it and have the parser fall back to the default text when it sees that.

As far as I have followed PW development, there is no alpha/beta stage, right?

The core on the master branch is always kept in a stable state, though individual modules may be kept in an alpha/beta stage. When the core is not in a stable state, I keep it in a separate dev branch (still on GitHub). Someday we may tie all updates to big versions, but for now, when something is ready to use or something is fixed, I like to make it available for people as soon as possible.

Another thought: Let's say I install a translation for PW 2.3 in a PW 2.2 installation (or vice versa). Are there going to be any "side effects"? Should we maintain versions of the translations for specific PW releases?

There shouldn't be any side effects. If any text in a translation changes, then it's considered abandoned, regardless of what version. If the text stays the same, then it's considered still good, regardless of version. The translations are tied to a hash of the default version as it appears in the PHP source files, so there's no potential of the wrong information to ever be sent via the translation. Version mismatches between the original and translation are technically impossible.

Then I update PW and the updated installation. In that case, all my additional translations will be overwritten by the updated translation, right?

Upgrades shouldn't touch your translations. This is one of the reasons why translations are kept as part of your /site/ structure rather than your /wire/ structure.

[yellowled]

I could maintain a default translation pack on GitHub, which already has all the files in it (empty, ready for edits).

While I do think this would help, a more comfortable inteface for editing and maintaining language packs in the backend might help to get PW translated to even more languages. I also assumed that the moved translations would be an exception. Generelly, I think PW will benefit from anything which doesn't add to your workload, so maybe the master translation isn't really necessary if it's a lot of work.

This one may be easy for us to accomplish if we just say that a translation with nothing but a period "." in it might be considered intentionally empty, or something like that. Then I can add a bit of code to detect it and have the parser fall back to the default text when it sees that.

Sounds good to me.

Upgrades shouldn't touch your translations. This is one of the reasons why translations are kept as part of your /site/ structure rather than your /wire/ structure.

Hm. Maybe something different went wrong in my case. Can't really reproduce it, so I was probably wrong about it. Sorry for making a fuzz about it.

The real "issue" at hand is, as soma points out, that once a translation hits a certain level of completion, it's very hard to keep it up to date on a regular basis. This may be my Kraut efficiency talking, but while I think it's great to have many translations, it doesn't really help if they're incomplete or outdated. In most open source projects, translations are done by non-coders, which is a great way for them to "give something back", also taking some work off of the developers. So translating should be as easy as possible. (It's not that I am lazy or something. )

[ryan]

While I do think this would help, a more comfortable inteface for editing and maintaining language packs in the backend might help to get PW translated to even more languages.

Are you thinking of something like a "find translatable files" tool that's built in? It could be done, but thinking it would be slow as it'd have to load up potentially thousands of files and scan them for translation functions… it can only scale so far. Whereas a quick grep at the command line can do it a lot quicker. Still, maybe it's worthwhile, as it would be nice to click on the file you want to translate rather than copy/paste the filename into the add file box. Plus also nice to see at a glance which files are new and don't yet have anything translated.

The real "issue" at hand is, as soma points out, that once a translation hits a certain level of completion, it's very hard to keep it up to date on a regular basis. This may be my Kraut efficiency talking, but while I think it's great to have many translations, it doesn't really help if they're incomplete or outdated.

Well we don't have to worry about out of date translations because the translation gets abandoned if the original changes. But it's true that incomplete translations are always possible and expected. But this is the nature of the beast, anywhere that translation is done, isn't it? When ProcessWire grows, new translatable copy comes into existence. I can't think of any solution to that problem other than stopping development?

Also want to mention that this is one reason why all of ProcessWire's translation tools are dynamic and built-in to the admin control panel itself. If you see something that's not translated, you can get to it with a few clicks and type in the phrase you want it to translate to. Compare that to most any other open source CMS (WordPress, Drupal, etc.) where the translations are handled with GNU gettext and pretty much out-of-reach of the client's reach or understanding. Those translations are dependent upon a 3rd party translator and developer. Whereas in ProcessWire, when a client sees some text that's not translated, and that they want translated, all the tools to do it are right there, ready for them to use.

[Pete]

Stop the development please, I want to get off

This reminds me that I really need to start thinking about translations when I do development myself, so I'm making a mental note and following these topics with interest.

[yellowled]

First of all, Pete is completely right: translation/internationalization is hard. It's tedious, nobody likes to do it, it sucks.

Are you thinking of something like a "find translatable files" tool that's built in?

I think I already mentioned that I'm involved in another open source project, which has very few developers, so we have to rely on non-developer help. Most of that is done in the translation area. From that experience, I would recommend to make creating and maintaining translations as easy as possible, even for "normal" users. It will help spread PW even more if it is properly translated in as many languages as possible.

Personally, I would be fine with the grep command. However, a lot of people have to work on servers or in small hosting plans which don't offer shell access. For those people, some kind of non-shell solution would be helpful. Doesn't have to be integrated in PW or the admin backend at all, it could as well be an extra PHP script or something. Just a solution to get the translatable files without shell access, maybe marking "new" translatable files, if that's possible.

I have to agree that PW's internationalization already is pretty comfortable. I also think it's perfectly acceptable to have some parts which are not (yet) translated. In my opinion, you can not use most open source software without knowing at least some English. Seems impossible.

But once PW starts to attract the "non-technical users" (which, as far as I can tell, it already is), there will be more people wanting to use it which barely understand English. Or not at all. Trust me, it will not keep them from wanting to use it anyway. You'll very soon get inquiries whether we could have a subforum in Spanish or something.

[Ralf]

...

You'll very soon get inquiries whether we could have a subforum in Spanish or something.

.. or German 8)

[yellowled]

Just to make this clear, I don't think this forum should have subforums in any language other than English.

[ryan]

Just want to make sure I've got it written down as to what specifically would help:

• Addition of an option to support intentionally-empty translations, and not have them count towards the untranslated count.
  
• Addition of a tool that finds all translatable files and also calls out those that don't have any translations (i.e. new files)
  
• Addition of a tool that bundles all translation files into a ZIP or something like that, ready for download.
  

Does this sound right, or anything else? I can't get started on these right away, but thought it would be good to get them written down so they can become part of the roadmap at least.

[Soma]

Great Ryan!

Question. What is the intended way to update a language pack's files? Drag them into clearly doesn't. Just upload them to the ftp folder?

Another thing that comes to thought, would it be to be possible to present core/system translation separate (just visually, tabs ...)? Not sure if that would help. Or maybe just filter them by /wire/ and /site/.

I like the way translation is an open and flexible system, just think any even so little assistance in maintaining it will be greatly appreciated. Thanks Ryan.

[yellowled]

I guess we could skip the zip functionality. Every hosting environment should offer some kind of possibility to get those files manually, at least via FTP.

Also, I think all of this can easily be a mid-to-long-term project. None of this is that crucial.

[ryan]

Question. What is the intended way to update a language pack's files? Drag them into clearly doesn't. Just upload them to the ftp folder?

Delete the ones you want to replace and drag them into the file field. I suppose you could also replace them via FTP, but you wouldn't want to do that for new files. I understand it would be ideal to just be able to drop and replace in 1 shot, but don't have an easy answer for that yet.

Another thing that comes to thought, would it be to be possible to present core/system translation separate (just visually, tabs ...)? Not sure if that would help. Or maybe just filter them by /wire/ and /site/.

I think they would have to be split into two separate file fields to do this. Might be a good thing to add during one of the next updates to LanguageSupport. But this would provide some separation between core and custom stuff, making it easier to maintain, which is what I think you are alluding to. Sounds good to me.

[Ivan Gretsky]

Great topic! And really nice to see that everything that has been outlined and planned got its way into reality.

I am trying to update russian translation right now and find it difficult to find the file with the string I want to translate. Am I completely missing the search and find capability, or is it not in place? If it is really not there, I think it would be great enhancement to find translatable string by text in default language.

[lisandi]

Much better would be to use a professinal tool for translatins and keeping the language files separate.
https://processwire.com/talk/topic/3317-get-localization/?p=79537