Progress on ProcessWire 2.2 and overview of multi-language support

[ryan]

I put in a lot of Language Support updates today, and put in the module dependency support while I was at it. One of the bigger things added was the multi language text fields. You can create new fields that using TextLanguage or TextareaLanguage (rather than Text or Textarea). You can also convert existing text fields to/from the multi-language versions. The multi language 'title' field isn't yet ready, but close. I also haven't yet put in a lot of testing into these multi language fields, so any help testing is appreciated.

Before you grab this update, I recommend uninstalling the LanguageSupport modules from your existing install (if you have them), or starting from a fresh install before testing. Also, as always, don't use the dev branch in production. Lastly, the TextareaLanguage field doesn't yet work with TinyMCE yet. It's probably a simple fix, but I ran out of time to work on it today so should have an update there soon.

I currently have the find() selectors designed to match values in either the current language OR default language. So if your 'summary' field is a TextLanguage field, Finnish is your current language and you execute something like this…

$results = $pages->find("summary*=Antti"); 

…It's going to match pages that have "Antti" in the Finnish summary or Default language summary field. But it won't match in any other languages, unless you set one of them to be your language. I figured this is the behavior that most would want, but can also make it match only in the Finnish summary (or make it configurable per-field).

[Soma]

I finally got a little more testing with the new admin language modules.

I really think it a great solution, and it's now easy to install and manage.

Now also checked out the new language text fields. Thanks for those. They can be very helpful I guess. But I question wether it's good to have them in the same fieldset or better using tabs. That way it would be more separated for editors. Or it may would be cool to allow seeing only certain language fields by a user. Just thinking out loud.

One thing I experienced was when changing the fieldtype to TextareaLanguage, and save it, the Label/Descr texts entered in the secondary language gets cleared. So I have to reenter them. When I switch back to Textarea.. they again get cleared.

[ryan]

Soma, thanks for testing this out.

Now also checked out the new language text fields. Thanks for those. They can be very helpful I guess. But I question wether it's good to have them in the same fieldset or better using tabs. That way it would be more separated for editors. Or it may would be cool to allow seeing only certain language fields by a user. Just thinking out loud.

For the purposes of translation consistency between languages (especially with the default language), I think it's good to be able to see them all at once ... at least as a default. But I think it will depend on the use. Right now I also question how useful multi language fields will be in practice, especially since we recommend multi-language trees instead. But it's also one of those things we can't do without because we need page references to have language support. Otherwise, using page references would be a lot more work. But because I'm not yet sure how much people will be using these multi language fields beyond this, I think it's best to start simple with the UI, and build from it as the practical uses become more clear. An option for tabs and per-user are good ideas that I'm sure we'll implement if people start using these a lot.

One thing I experienced was when changing the fieldtype to TextareaLanguage, and save it, the Label/Descr texts entered in the secondary language gets cleared. So I have to reenter them. When I switch back to Textarea.. they again get cleared.

I'm having the same issue-it's a bug. Should have it cleared up soon.

Thanks,

Ryan

[ryan]

All of the language features are now in place in the dev branch and I think we're at a good RC1 release with PW 2.2. Here's a short video intro on how to use the multilanguage fields (though I'll plan to make a better one next week).

http://processwire.com/pw-multilang.mov

[Oliver]

I’m currently setting up PW with LanguageSupport for a client project. And so I experimented with the $this->_() thing trying to switch the whole backend to German, as the client wants it this way.

Working on this two issues came up:

First one is the aready known getModuleInfo problem, as it is used to display infos on all available module admin interfaces in the backend’s setup area. I think that making this part translatable is a crucial point in view of this list being the user’s gateway to more advanced functionalities provided by – I’m pretty sure – more and more available and extensive modules.

Another one came up when I was trying to make the translator itself translatable. (Why isn’t it translatable by default?)

When the following is used in modules

		$this->set('showFields', array('name', 'title', 'language_files'));

you can’t translate the table’s column names without corrupting the list rendering.

I really love the system languages feature so far, anyways.

[ryan]

Oliver thanks for testing out the Language modules and for your feedback.

First one is the aready known getModuleInfo problem, as it is used to display infos on all available module admin interfaces in the backend’s setup area. I think that making this part translatable is a crucial point in view of this list being the user’s gateway to more advanced functionalities provided by – I’m pretty sure – more and more available and extensive modules.

The initial release of PW22 focuses on making the client-side stuff translatable. Ultimately all the developer-side stuff (like Setup and Modules) will be translatable too, but I consider that secondary. I will be focusing on a module or two every week for translation and adding them to the repository. Another factor here is that there is a large amount of translatable stuff in the developer-side part of PW, relative to the client (Pages) side. I estimate there's 3-4 times as much translatable copy in that stuff. So in the short term, I want to make sure that everything your client sees is in their language. Then we'll focus on all the stuff that we–the web designers/developers–see.

When it comes to the Modules section, we'll make the stuff in getModuleInfo() translatable for the core modules. It'll be up to other module developers whether they want to make their modules translatable, but it'll be recommended.

$this->set('showFields', array('name', 'title', 'language_files'));

you can’t translate the table’s column names without corrupting the list rendering.

I'm not sure that I understand what you mean? At least, I'm not clear about this example/where it's from?

But want to mention that table column names (and field names) will not be translatable–when it gets down to that level, we need to have something concrete in order to relate to the database schema. Of course, field labels and descriptions are translatable though.

[Oliver]

Ok, so I’ll try to keep the client out of the module and setup section for now.

Sorry, my code example really wasn’t very enlightening. Of course I know we deal here with the actual field names of the database schema and the relation would be lost as soon as I change the showFields array. My thought just was, that in view of usablity/readability aspects there should be a way, to change the table’s column headers without leaving the fieldname thing. For example:

$this->set('showFields', array('name'=>$this->('Name'), 'title'=>$this->_('Language'), 'language_files'=>$this->_('Language Files')));

as an optional alternative to

$this->set('showFields', array('name', 'title', 'language_files'));

A similar approach like you already implemented in your »row()«-syntax:

$table->row(array(
                 'text' => 'link',
                 'only text'
);

Just an idea.

But there is another point. As soon as $this->_() is used multiply within a command like this

$this->method(array($this->_('Text1'), $this->_('Text2')));

or this

$field->value = "<span>".$this->_('partoftext')."</span> ".$this->_('another part of the text');

just the first $this->_()-phrase of the line seems to be displayed for translation in the LanguageTranslator.

[ryan]

My thought just was, that in view of usablity/readability aspects there should be a way, to change the table’s column headers without leaving the fieldname thing. For example:

Now I understand what you mean. I've just updated the source so that now it uses the field labels rather than the names. That way it'll pull in the field label in the language that it's in. So for the examples you posted, you'd want to edit the language and language_files fields in Setup > Fields > Filters > Show Built in Fields. I've also made ProcessPageType translatable so that everything else in that file can be in future language packs (including the 'Name' field, in the column headers).

just the first $this->_()-phrase of the line seems to be displayed for translation in the LanguageTranslator.

That's a good observation. There are a couple rules set by the parser that vary from regular gettext. One is that there can only be one translation per line. Another is that a translation block cannot contain split strings, and thus cannot span more than one line (though you can put literal "\n" returns in it if you want). So to achieve the example you posted, you'd need to do this:

<?php
$this->method(array(
    $this->_('Text1'), 
    $this->_('Text2')
    ));

or this:

<?php
$field->value = "<span>".$this->_('partoftext')."</span> ".
    $this->_('another part of the text');

or this:

<?php
$a = $this->_('part of text'); 
$b = $this->_('another part of the text'); 
$field->value = "<span>$a</span> $b";

The reason for this is that the parser considers comments on the same line as the translation to be notes for the translator:

<?php
$this->method(array(
    $this->_('Text1'), // This will show up as an optional label for the translation field 
    $this->_('Text2')  // Some label // a second comment on the line adds an extra 'notes' field visible to translators
    ));

[formmailer]

All of the language features are now in place in the dev branch and I think we're at a good RC1 release with PW 2.2. Here's a short video intro on how to use the multilanguage fields (though I'll plan to make a better one next week).

http://processwire.com/pw-multilang.mov

Hi Ryan,

Pretty cool that you used the Dutch translation in the movie.

When looking at the movie, I started wondering how the languages work for guest users. Registered users can, of course, select there language in the profile, but guests can't do that.

Can a guest look at the site in another language than the default?

/Jasper

[apeisa]

Actually forget to mention this earlier - it seems that quests don't see the default language, they see always __("Original tags"). Not sure if this is already fixed, since that was few iterations ago...

[ryan]

Actually forget to mention this earlier - it seems that quests don't see the default language, they see always __("Original tags"). Not sure if this is already fixed, since that was few iterations ago...

This one should already be fixed (just tested to confirm).

Pretty cool that you used the Dutch translation in the movie.

Thanks for the translation!

When looking at the movie, I started wondering how the languages work for guest users. Registered users can, of course, select there language in the profile, but guests can't do that.

Can a guest look at the site in another language than the default?

It just depends how you code the site. You'd have to set their language, like this:

$user->language = $languages->get('dutch'); 

So you might do that if you'd set a value to $session->language, or if $input->urlSegment1 == 'nl' or something like that. Lots of possibilities.

[formmailer]

It just depends how you code the site. You'd have to set their language, like this:

$user->language = $languages->get('dutch'); 

So you might do that if you'd set a value to $session->language, or if $input->urlSegment1 == 'nl' or something like that. Lots of possibilities.

Thanks. I should have known it would be that easy.

I just added ProcessPageType to the Dutch and Swedish translations (on Github).

Ryan, is there a way to see a list of all translatable files available, so we can check if we have all of them translated?

/Jasper

[Oliver]

Just another thing I stumbled upon:

In the admin area you get the same view under /processwire/page/ as under /processwire/page/list/. But while on /processwire/page/ the translations of the admin default template replace the original text, on /processwire/page/list/ the template translation is still used, but the page title »Page List« is still in English and isn’t listed for translation in LanguageTranslator anywhere. Neither in wire--templates-admin--default-php.json, nor in wire--modules--process--processpagelist--processpagelist-module.json. Same thing with the titles of the html document.

Having a look at the corresponding code sections in the admin template

<title><?php echo __(strip_tags($page->get("browser_title|headline|title|name")), __FILE__); ?> • ProcessWire</title>
…
		<h1 id='title'><?php echo __(strip_tags($this->fuel->processHeadline ? $this->fuel->processHeadline : $page->get("title|name")), __FILE__); ?></h1>

it seems to me that they are meant to be translatable. Right?

[ryan]

In the admin area you get the same view under /processwire/page/ as under /processwire/page/list/. But while on /processwire/page/ the translations of the admin default template replace the original text, on /processwire/page/list/ the template translation is still used, but the page title »Page List« is still in English and isn’t listed for translation in LanguageTranslator anywhere.

Good find Oliver! I hadn't realized that, but you are right, it's using a different Page Title depending on what URL you access it from. I've just pushed an update that ensures that page title is consistent so that it doesn't have to translate it twice. It should now always be 'Pages'. Thanks for finding this.

<title><?php echo __($page->get("title|name")); ?> • ProcessWire</title>
…
<h1 id='title'><?php echo __($this->fuel->processHeadline ? $this->fuel->processHeadline : $page->get("title|name")); ?></h1>

I'm actually pulling a bit of a trick there to enable dynamic page titles like 'Pages' and 'Setup' to be part of static language packs. The above __() function calls won't register with the parser because they have no static text defined in them. However, I know that there are certain words like 'Pages' and 'Setup' that are part of almost every PW installation, and I want them translated when they happen to show up. The comment near the top of the file helps to explain it:

/*
* Dynamic phrases that we want to be automatically translated
*
* These are in a comment so that they register with the parser,
* in place of the dynamic __() function calls with page titles. 
* 
* __("Pages"); 
* __("Setup"); 
* __("Modules"); 
* __("Access"); 
* __("Admin"); 
* 
*/

Those function calls in the comment above are seen by the parser but not by PHP. So the end result is that if they appear by themselves in the <title> or <h1> tags above, they will get translated automatically.

This method isn't strictly necessary, as we could have done it with multilanguage page title fields. But since translations in multilanguage fields aren't part of language packs, this was a way of ensuring they could be part of language packs. 

I just added ProcessPageType to the Dutch and Swedish translations (on Github).

Ryan, is there a way to see a list of all translatable files available, so we can check if we have all of them translated?

Thanks for the update Jasper! Here is a list of all the current translatable files. I will plan to add an official and always up-to-date list to our language translation documentation (currently in production). This list will continue to grow over the coming weeks.

Inputfield Modules

/wire/modules/Inputfield/InputfieldEmail.module

/wire/modules/Inputfield/InputfieldFile/InputfieldFile.module

/wire/modules/Inputfield/InputfieldImage/InputfieldImage.module

/wire/modules/Inputfield/InputfieldPageListSelect/InputfieldPageListSelect.module

/wire/modules/Inputfield/InputfieldPageName/InputfieldPageName.module

/wire/modules/Inputfield/InputfieldPassword.module

/wire/modules/Inputfield/InputfieldSubmit/InputfieldSubmit.module

/wire/modules/Inputfield/InputfieldTinyMCE/InputfieldTinyMCE.module

/wire/modules/Inputfield/InputfieldURL.module

Language Support Modules

/wire/modules/LanguageSupport/LanguageSupport.module

/wire/modules/LanguageSupport/ProcessLanguageTranslator.module

Process Modules

/wire/modules/Process/ProcessForgotPassword.module

/wire/modules/Process/ProcessLogin/ProcessLogin.module

/wire/modules/Process/ProcessPageAdd/ProcessPageAdd.module

/wire/modules/Process/ProcessPageClone.module

/wire/modules/Process/ProcessPageEdit/ProcessPageEdit.module

/wire/modules/Process/ProcessPageEditImageSelect/ProcessPageEditImageSelect.module

/wire/modules/Process/ProcessPageEditLink/ProcessPageEditLink.module

/wire/modules/Process/ProcessPageList/ProcessPageList.module

/wire/modules/Process/ProcessPageSearch/ProcessPageSearch.module

/wire/modules/Process/ProcessPageSort.module

/wire/modules/Process/ProcessPageType/ProcessPageType.module

/wire/modules/Process/ProcessProfile/ProcessProfile.module

Admin Theme

/wire/templates-admin/default.php

[Nico Knoll]

Why is your parser that complicated. I mean I also used a __() function for translations month ago and it always worked like this:

function __($string,$type) {
global $db, $lang;

if(is_file('languages/'.$db->get_element('setting','value','`key`=\'language\'').'.php')) {
	require_once('languages/'.$db->get_element('setting','value','`key`=\'language\'').'.php'); // something like de_DE.php
}

if(isset($lang[$type][$string])) {
        return $lang[$type][$string];
    } else {
        return $string;
    }
}

So you could e.g. have multiple translations in one line, ...

/ Nico

[ryan]

Why is your parser that complicated. I mean I also used a __() function for translations month ago and it always worked like this:

We're duplicating GetText functionality here, not just a string translator. Everything about it is geared towards making it super-simple to use, and able to scale to any size. I actually think we have a solution that is significantly less complex than GetText, while duplicating its functionality. For example, take a look in the WordPress i18n documentation and read the sections for: POT files, PO files, MO files, "Choosing and loading textdomains", and "Generating POT files". Note that you don't have to deal with any of that in ProcessWire, nor do you have to install GNU Internationalization utilities, nor do you have to install and use POEdit. Yet you get all the same capability and power–ProcessWire's Language Support does it all for you. If you want to compare ProcessWire's implementation, compare it to another of the same capabilities like in WordPress or Drupal (both of which use GetText). I think you'll find ProcessWire's implementation is far simpler and just as powerful from every standpoint.

The example you posted doesn't involve a parser. By that, I mean a parser like GetText that sifts through PHP files and locates the translatable phrases on it's own, outside of program execution. Without a parser, you have to predefine everything on your own in a separate PHP file (or some other file), like in your example. This is okay for small things, but we need something more powerful and scalable than that. While there's a lot of power under the hood, I think our basic translation API is about as simple as it gets:

<?php
echo __("text"); // outside of a class
echo $this->_("text"); // inside of a class

With regard to number of translation function calls on a line, there's no problem with placing multiple __() function calls on the same line from a PHP perspective–all those functions will still get called at runtime. It all has to do with the external parser (like GetText, or in our case PW's LanguageParser.php). We need to have a way to let the parser identify the translatable phrases outside of PHP while also optionally communicating additional meta data about the translation. This part happens at translation time, not at runtime, so that everything is cached and ready to go with no parsing at runtime. The 1-translation per-line requirement is to ensure that optional meta data about the translation can be communicated to the parser (and people doing translations), and that meta data is communicated via comments placed at the end of the line. Since this meta is optional, it's feasible we could just ignore it when there are multiple translations on a line (and perhaps will add that), but for now I'm trying to keep it simple with clear guidelines and best practices.

It goes beyond technicalities or rules too. Keeping translations on one line helps to keep things more readable and clear (for me at least, having now made dozens of files translatable). Language designed for translation is different from regular language in that it requires more careful consideration. I've had to rethink my use of English in many cases. Due to the differences in languages, translatable phrases are by nature independent pieces of information that must always be static and carefully considered, especially when it comes to how it flows with other dynamic text. Note that GetText (WordPress/Drupal) has the same requirement when it comes to providing meta data to a translation function call. However, they also don't prevent you from putting several calls on one line (with no meta data) so we may update to do the same if there is enough demand for it.  But I'd still encourage people to follow these rules even if we just make them optional guidelines.

[ryan]

I've posted full documentation for how to use ProcessWire's translation features in your own code:

http://processwire.com/api/multi-language-support/code-i18n/

[ryan]

Just posted full documentation on multi-language fields, language-alternate fields, and how to use them. Also includes examples of how to use language fields to create full multi-language websites. Examples include using subdomains, URL post-segments, URL pre-segments, and sessions. I'm interested in your feedback.

http://processwire.com/api/multi-language-support/multi-language-fields/'>http://processwire.com/api/multi-language-support/multi-language-fields/

I'm also working on more additions to the new multi language support section of the documentation (http://processwire.com/api/multi-language-support/).

Thanks,

Ryan

[apeisa]

Absolutely brilliant stuff Ryan!

[Soma]

Awesome stuff Ryan as always. Interesting read for sure! Thanks so much for this.  ;D

[SiNNuT]

I won't be needing a multi language site anytime soon but it's great to see PW having all the bases covered here. Lot's of options.

[achabany]

Hello, i am new to the forum, first i want to thank you ryan and all of you guys for this great app i have just found an issue with the translation system : the breadcrumb for the "users", "roles" and "permissions" will turn you every time to the access page wich is not correct. The problem come from this pageType module :

public function init() {
 $this->config->scripts->add($this->config->urls->ProcessPageType . 'ProcessPageType.js');
 $this->config->styles->add($this->config->urls->ProcessPageType . 'ProcessPageType.css');
 $this->fuel->breadcrumbs->add(new Breadcrumb('../', $this->page->get('title|name')));
 $this->pages = $this->fuel($this->page->name);
 if(is_null($this->pages)) $this->pages = $this->fuel('pages');
 if($this->pages instanceof PagesType) $this->template = $this->pages->getTemplate();
 parent::init();
}

I removed this line in the init function :

$this->fuel->breadcrumbs->add(new Breadcrumb('../', $this->page->get('title|name')));

And I have placed this line in the ___executeEdit() and ___executeAdd() functions :

$this->fuel->breadcrumbs->add(new Breadcrumb('../', $this->page->get('title|name')));

Hope this help

[Soma]

Thanks and welcome to the forum!

I think the problem is that the breadcrumb shows the current page. like "Users" at the end, but it shouldn' even show up. It should be like on setup sub pages, it doesn't show "Fields" for example if you're on the "Fields" page.

[ryan]

Thanks for the issue report Achabany--I will correct this like you mentioned.

Also, thanks for the work on the French language pack. I agree about setting up a board just for translations. I'm still learning the ropes of IP.Board but hope to have that setup this wekend or early next week.

[jiserra]

Hi! My name is Juan Ignacio and I'm from Argentina. I was really expecting the 2.2 release because finally I can use the CMS on my clients sites! The only thing that keep moving me from WP is the admin language support and now is finally here! I think processwire is really great, congrats to all the team!

I've read this thread and maybe I'm missing something but I couldn't find the Spanish Language Pack on the subforum, it exist? I know that Ryan did a Spanish pack but it was months ago and maybe is outdated? I'm happy to contribute in the translation if the pack is not ready