Fluency - Integrated DeepL Powered Content Translation

[Jozsef]

Thank you @FireWire, after some wrestling I got it working. My issue was that I assumed that the returned translation is a simple string while it's not.

For anyone interested, this is the working function I put together. It returns the translated field value or translates a multi-language field (headline in this case) on demand, if it's not yet translated.

You can call it simply <?= translate($page,'headline'); ?>. It's saved and displayed right away.
This way it can also be used in a foreach loop where you want to display a field from a different page.

Please let me know if something could be improved or simplified, I'm not a PHP guru.
Do you think it will have a toll on performance for already translated fields?

/* Translate a field if it's not available in current language
 *
 * @param object $page
 * @param string $field
 * @return string
 *
 */
function translate($page,$field) {
	// If field is empty there's nothing to be translated
	if (!$page->$field) return;
	// Get current language setting
	$lang = wire('user')->language;
	$page->of(false);
	// Check if the field is already translated
	$local_value = $page->$field->getLanguageValue($lang);
	if (!$local_value) {
	  $fluency = wire('modules')->get('Fluency');
	  // If not, translate it from default language
	  $translate = $fluency->translate('en', $page->$field, $lang->language_code);
	  // Get the translated string from the response object
	  $translated = $translate->data->translations[0]->text;
	  if ($translated) {
	  	// Save the translated value back to the field
	  	$page->$field = $translated;
	  	$page->save($field);
		// Set return value
	  	$local_value = $translated;
	  } else {
		// If translation fails and returns empty string
		// We use the default language
		$local_value = $page->$field;
	  }
	}
	$page->of;
	return $local_value;
}

There's still the issue of updated content in the original language but I'm not sure how to deal with that.

[FireWire]

Translating only for missing content should be good for performance. I do think that there are some items to consider.

• If this is used in a template when the page loads and caching is used then this function isn't guaranteed to execute since a pre-rendered HTML document would be returned to the browser. Caching is a good idea for performance so this would create a situation where you can't use caching and that would be a performance hit.
• If the content has been changed but you're only checking for the existence of translated text then it wouldn't re-translate. I have a solution for this but it would take a little extra code (detailed below)
• There is a possibility that using the function you wrote could mean additional calls to the database. Someone with a little more knowledge of the ProcessWire core could correct me if I'm wrong. If that's the case then there could be a performance hit that is dependent on how performant your database setup is. Performance difference would depend on how performant your DB and DB connection is.
• If you are running this loop on page load then the performance hit would really come from the delay in response from DeepL. Translation can take a couple of seconds in some cases and that will slow your page load time down a lot. This would only happen as long as something needs to be translated. If not then it will skip over the field and the page will load.

As for the translation return value- the module returns a passthrough payload directly from the DeepL API (I didn't develop the return data structure). This is good because it is predictable and unchanged from DeepL documentation, and it makes sense when you consider the ability to translate multiple separate texts at once. Take some time to review the README.md file in the Fluency module directory, it has documentation of the return data structure and details on using the module directly. I'll make a note to let module users know that information is there for review.

Solution for tracking changed content:

One way to verify content is to use hashing and WireCache. I am developing the next version of Fluency that will have a solution for this but in the meantime here is a modified solution that may fit your use case:

<?php

/**
 * Analyzes the content in a field determines if it has changed since the last time it was checked
 * @param  Page      $thisPage  Page containing field to check
 * @param  string    $fieldName Field object to check content for
 * @param  string    $lang      Name of language to check content for default is PW's default language
 * @return bool|null            Bool for content change, null if field doesn't exist on page
 */
function fieldChangedSinceCheck(Page $thisPage, string $fieldName, string $lang = 'default'): ?bool {
  if (!$thisPage->hasField($fieldName)) {
    return null;
  }

  $thisPage->of(false);
  $pageField = $thisPage->fields($fieldName);
  $isMultilanguageField = $pageField->type instanceof FieldtypeLanguageInterface;

  // Handle multi-language field
  if ($isMultilanguageField) {
    // Get the language, content in that language, and create a unique tracking ID
    $language = wire('languages')->get($lang);
    $current_field_content = $thisPage->$pageField->getLanguageValue($language);
    $key = "{$thisPage->id}|{$pageField->id}|{$language->id}";
  }

  // Handle non multi-language data
  if (!$isMultilanguageField) {
    $current_field_content = $thisPage->$fieldName;
    $key = "{$thisPage->id}|{$pageField->id}";
  }

  // Create a unique hash for the current content in this field in this language on this page
  $current_field_content_hash = hash_hmac('sha256', $current_field_content, $key);

  // Search WireCache for a previously stored content hash under this key if it exists, otherwise null
  $cached_field_content_hash = wire('cache')->getFor('field_content_tracking', $key);

  $contentHasChanged = false;

  // Compare the hash we created for the current content and compare it with the hash previously stored
  // If they do not match either it does not exist, or the content has been changed since it was last
  // analyzed
  if ($current_field_content_hash !== $cached_field_content_hash) {
    // Store the current content hash which will be used later to compare if content has changed
    wire('cache')->saveFor('field_content_tracking', $key, $current_field_content_hash);

    $contentHasChanged = true;
  }

  return $contentHasChanged;
}

There is a caveat, this solution will only tell you if it has updated since the last time it checked. So if a field was changed and you check the function will return true. If you check it again it will return false because it only tracks if it has changed since the last time it checked. I think this will still work for your use case as long as you act on it when it returns true. It works with multilanguage fields, regular fields, any language, and returns null if you try to check a value for a field that isn't on the page. Hope it helps!

[Jozsef]

Thank you for the detailed answer, it's very educational. I was aware of some of the considerations but I didn't think about cache, for example.
I understand that the DeepL service take time, especially with a 5000 word document. ?
I did go through the readme multiple times before.
Great module and thanks again for the brilliant support here.

[ngrmm]

@FireWire thanks for the module!
Is there a way to force regular mode for CKEditorfields inside tables and use fluency? Or are they always in inline-mode, no matter what setting are set?

[FireWire]

6 hours ago, ngrmm said:

@FireWire thanks for the module!
Is there a way to force regular mode for CKEditorfields inside tables and use fluency? Or are they always in inline-mode, no matter what setting are set?

I haven't experienced that. I don't often use tables and can't remember if I've seen that.  Fluency wouldn't be able to change how a field is rendered. 

[ngrmm]

1 hour ago, FireWire said:

I haven't experienced that. I don't often use tables and can't remember if I've seen that.  Fluency wouldn't be able to change how a field is rendered. 

Looks like it's not possible. It says:

• settingsField: Specify a PW field that uses CKEditor to use for the settings of this column. Note that inline mode is always used, regardless of field settings.
• …

 

[FireWire]

5 hours ago, ngrmm said:

Looks like it's not possible. It says:

• settingsField: Specify a PW field that uses CKEditor to use for the settings of this column. Note that inline mode is always used, regardless of field settings.
• …

 

Dangit. Sorry to hear that. The inline CKEditor thing is on my list but won't be added until the next version which I'm working on right now. The way that field renders is much different from other fields. I'll be sure to add that to the priority list to see what can be done.

[jacmaes]

@FireWire I imagine that TinyMCE is a whole different beast and would require a ton of refactoring, but I was wondering whether you have considered working on a version that would support Processwire’s new default editor. Thanks for your work!

[FireWire]

On 5/13/2023 at 3:23 PM, jacmaes said:

@FireWire I imagine that TinyMCE is a whole different beast and would require a ton of refactoring, but I was wondering whether you have considered working on a version that would support Processwire’s new default editor. Thanks for your work!

I've been following the news on the new editor and I'll be adding that to the next version of the module. The module is already getting refactored so I'll be able to plan for this much more easily.

I'll work on posting some updates here with some details on what's planned!

[FireWire]

Hello all!

I've compiled a list of features that Fluency will have on it's next release and currently under development. There are a lot of new features and, as mentioned, it's being completely rewritten. I want to share some notable features that I think will be some great additions. IMHO, I believe that this puts Fluency in a first-rate position to make ProcessWire easier and more powerful for multi-language sites compared to other CMS/CMF platforms. If you've tried to use translation modules/plug-ins for other platforms I think you'll agree.

As you can imagine, the feature expansion makes the module very powerful but it also means a lot of extra work. There will be an upgraded version available named (unsurprisingly) FluencyPro. Before I get into that, my commitment is to make the non-pro version the real-deal when it comes to quality and features. No features will be taken away from the core Fluency module and moved to the premium module. As you'll see below, the majority of new features will be available in the Fluency core version. Every feature in FluencyPro is new and complements the Fluency module that is and will always be free. I also want to make the Pro version very reasonably priced so it can remain within reach for as many people as possible and an easy sell to clients. The price will help offset the time and effort it takes and the support is greatly appreciated. More details to follow on that later.

I chose the new features based on feedback from the community as well as my usage for sites I've built.

Here is a list of new features that are coming in the next version:

Fluency (core)

• Continuity - All existing features will still be available
• Localization - All Fluency admin UI elements can be translated to all languages present in ProcessWire. Includes the translation trigger buttons on each field, the global translation tool in the menu, etc.
• Error Handling - Will indicate things like the DeepL service not being available, usage limit reached. All errors will be translatable.
• Improved Module Config - Cleaner and more organized module config screen
• Inline Fields - Support for inline fields
• TinyMCE - Support for the new rich text editor
• CKEditor - Continued support for CKEditor to keep the module backwards compatible with ProcessWire sites that don't/can't use TinyMCE
• Per-language change indication - When content is changed in a field, the corresponding language tab will indicate that the content has been changed. This makes it easy to see where content should be updated in other languages.
• Translation Caching - Translations will be cached so that additional requests for the same content will not require additional DeepL API calls. This can help keep monthly DeepL API usage lower where possible and make repeated translations lighting fast. Cache can be manually cleared in the module config screen
• Logging - Improved
• Remains free - Free forever and will be open-sourced

FluencyPro

• Multi-Language - Translate any content to any language for a field.
• Multi-Language - Translate any content from any language for a field.
• One Click Translate All - Any language to every other language for a field.  This makes using a wider array of languages trivial when editing pages
• Markup Companion Module - Optional FluencyMarkup module provides individual easy-to-use methods for markup output to the front-end. This makes Fluency a complete translation solution out of the box. Features:
  • Render all languages as prebuilt `<a>` links to navigate between page languages.
  • Render a self-contained prebuilt `<select>` element with one click switching between languages on the front end while needing no additional JS required in your code. Inline JS is optional for those who prefer to implement their own.
  • Render all alternate language `<link rel="alternate" hreflang="{ISO code}" href="{alt language URL}">` `<head>` tags to indicate that the page has additional versions in other languages. Great for SEO and adhering to HTML standards.
  • Output the current language ISO code. Useful for turning `<html>` into `<html lang={ISO code}>` in keeping with HTML best practices.
  • All render methods are hookable for additional customization by developers

Notable overall code improvements:

• Written using PHP 8.0 (required for use)
• JavaScript rewritten in ES6 to make use of newer language features, is transpiled to ES5 to maintain browser compatibility. Transpiling is pre-configured and included with the module.
• Modular per-inputfield JavaScript so that adding new field types and updating existing ones in the future is faster and easier. Standardized composition so interfacing with fields within code is uniform and predictable.
• Server-side module code is being rewritten with the future potential to add additional "translation engines". While there won't be additional services available now, there will be a more standardized interface to make adding others easier. Both versions of Fluency will have this feature and can benefit from new translation services other than DeepL. This won't be 100% ready on release, but it is being kept in mind now.
• Future translation engines will have access to a uniform interface to use the Fluency caching feature with no additional overhead during development.
• Improved RESTful style API within the admin which lets other modules or PW customizations implement separate translation features using AJAX requests
• Leverages ProcessWire's built in JavaScript config object (as opposed to a separate API request on page admin page load) to speed up UI initialization
• .prettierrc and .editorconfig files to make contributing easier

Thanks everyone for your patience, really excited for the new features! Cheers!