Jump to content

Fluency - Integrated DeepL Powered Content Translation


FireWire

Recommended Posts

19 hours ago, adrian said:

@FireWire - just a reminder to remove the bd($content) call in Fluency.module.php 

Also, just want to agree with @xportde's request for a dedicated permission. I have already changed it locally to add a "fluency" permission. In my use case I actually needed to give the guest user this permission so that I could do translations via a cron called script.

Thanks again for your work on this - I am now using it in production.

Pushed the fix that removed the bd() call. Thank you for the reminder. I put some work in over the weekend on the permission feature. The new permission is named "fluency-translate" in case you want to rename yours to match the later releases. This release also formally adds the 5th Fluency->translate() argument to feed in additional API parameters.

Everything is always a time crunch between holidays here, looking forward to when I can get back to working on more changes. Permission will be the next feature.

Excellent to hear that it's in production! We've been using it and it's been smooth sailing.

  • Like 1
Link to comment
Share on other sites

@FireWire - not sure if you want to bother with this, but I just found out that DeepL doesn't yet support translating between US and UK English. I was hoping to be able to change things like neighbor to neighbour. I have contacted DeepL and asked them to consider supporting this. Strange thing is that you can actually translate from US English to Spanish to UK English, eg: neighbor -> vecino -> neighbour.

Hopefully they'll support direct translation in the future, but until then I wonder if you want to remove that option from the Translation process module?

  • Like 1
Link to comment
Share on other sites

On 12/9/2020 at 7:49 PM, adrian said:

@FireWire - not sure if you want to bother with this, but I just found out that DeepL doesn't yet support translating between US and UK English. I was hoping to be able to change things like neighbor to neighbour. I have contacted DeepL and asked them to consider supporting this. Strange thing is that you can actually translate from US English to Spanish to UK English, eg: neighbor -> vecino -> neighbour.

Hopefully they'll support direct translation in the future, but until then I wonder if you want to remove that option from the Translation process module?

That's interesting, I'd never actually tested that. I'll make a note to disable those translation connections in the module.

Also really odd that capability doesn't exist... seems like literally the easiest translation they'd have to do haha.

  • Like 1
Link to comment
Share on other sites

Important update to the module with a new release. Recommended that all users upgrade to the latest version

The module has been updated with critical fixes and feature requests fulfilled.

  • The module has been updated to fix an issue with lengthy content that would result in failure, mostly important for CKEditor fields that include markup in the content. This was due to the admin AJAX call to the module using a GET request where a POST request should have been used to prevent running into URL length issues.
  • Translation was pulling from the wrong location in CKEditors so it was translating the last saved content, not the latest changes made in the field before page save.
  • Use of Fluency is now permission based as requested. After upgrading all roles who should have access must have the fluency-translate permission added
  • The configuration screen now has a new setting for the Preserve Formatting feature which addresses the issue reported in this thread where short strings or phrases were being returned with additional/missing punctuation. (internally this uses the preserve_formatting DeepL API parameter). It is recommended that this be enabled unless you have a specific reason not to use it.
  • Public methods have been added to the Fluency module so that translation, usage stats, and available languages can be pulled from calling the Fluency module or by using the DeepL class directly. README has been updated with documentation for usage.
  • A new parameter has been added to both the Fluency and DeepL translate methods that allow you to pass in any DeepL API key/values with your request. If you are currently calling either the module or DeepL class directly, please read the documentation as the parameters have changed.
  • Return values from the Fluency module now match the return values from the DeepL module for consistency. This includes passing DeepL's HTTP response for use in programmatic conditions.
  • The module has been moved from Gitlab to Github to adhere to ProcessWire's module directory requirements for future listing.

As usual, more to follow! Latest: https://github.com/SkyLundy/Fluency-Translation/archive/main.zip

Paging @adrian & @xportde & @bartelsmedia as you've been Fluency's alpha testers extraordinaire.

  • Like 2
Link to comment
Share on other sites

  • 1 month later...

@FireWire - sorry for the silence on your last post - I haven't had a chance to test this new version yet - hopefully soon.

A question for you thought - have you looked at: https://lingvanex.com/translationapi/ - they are much cheaper than DeepL and support a LOT more languages. This post is also worth a read: https://lingvanex.com/history-lingvanex/

Anyway, I am need of Tagalog translation and am thinking about Lingvanex but would like to know if you investigated it and decided against it for some reason.

Thanks for your input.

Link to comment
Share on other sites

34 minutes ago, adrian said:

@FireWire - sorry for the silence on your last post - I haven't had a chance to test this new version yet - hopefully soon.

A question for you thought - have you looked at: https://lingvanex.com/translationapi/ - they are much cheaper than DeepL and support a LOT more languages. This post is also worth a read: https://lingvanex.com/history-lingvanex/

Anyway, I am need of Tagalog translation and am thinking about Lingvanex but would like to know if you investigated it and decided against it for some reason.

Thanks for your input.

Really no problem at all. I've released a newer version since then that addresses an issue that I didn't see come up in testing previously. Just a courtesy tag in case the bugfixes end up helping someone.

I haven't heard of Lingvanex before so I didn't know to look into them. Initially when I found DeepL I was really impressed by the service but needed someone else to really verify what was going on because I only speak English. I had a coworker who speaks Spanish look at the results and he was very impressed. I kind of stopped there and didn't think to look further. Overall if someone was able to verify the quality of translation of any translation then that's all that matters.

I don't have any personal allegiance to DeepL, it's just what I found and what I heard about on the PW forums which also nudged me towards them. It would be interesting in the future to update Fluency to allow the dev to select a translation engine, and the way the module is built I think it could be done well. I wish I could work on that now but I just had 3 clients contact me for work and that's in addition to my day job...

 

  • Like 2
Link to comment
Share on other sites

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

  • Like 2
Link to comment
Share on other sites

16 hours ago, adrian said:

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

Appreciate the consideration! I tried to make Fluency as modular as possible so hopefully it shouldn't be too hard to add another engine. If you take a look at how DeepL works with Fluency, it exists as it's own class and Fluency doesn't know how or what happens in DeepL, and vice versa. Following that structure, you could create a Lingvanex class that Fluency sends data to. Matching DeepL method parameters when calling it and data structures it returns could make it pretty straightforward.

An idea I had would be to create a bootTranslationEngine() method that is called in ready() which would determine what engine/API key is available and instantiate the proper translator class in something like $this->translationEngine. That would replace $this->deepL in the Fluency module. I would make it a goal to abstract out any references to any brand of translator in the Fluency module's logic.

As long as getClientBootData() and translate() take and return the same structure of data you wouldn't even need to touch the front end.

I haven't really written any code that is collaborated on before so I hope that the commenting and overall structure makes sense. If you ever have any questions or want a shortcut to figuring something I did out then message me and we can chat.

 

 

  • Like 2
Link to comment
Share on other sites

  • 1 month later...
On 2/19/2021 at 7:22 AM, adrian said:

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

HI @adrian, any news on Lingvanex PR? 

Link to comment
Share on other sites

  • 4 weeks later...
On 4/14/2021 at 10:02 AM, B3ta said:

HI @adrian, any news on Lingvanex PR? 

Sorry, because I don't actually use a translation service to translate content in the admin I took the easy way out and just implemented a direct curl request to the Lingvanex API.

Link to comment
Share on other sites

  • 4 weeks later...

Very nice module. It actually also works without a paid account using the free api. The only thing one needs to do is to change one line as the api endpoint is different. 500000 characters for free per month which should probably be enough for many applications. 

classes/DeepL.class.php:    $reqUrl = "https://api-free.deepl.com/v2{$endpoint}?auth_key={$this->apiKey}";

 

I got a javascript error with the module though if I use a non-CK Editor text field (CK editor not found). The textarea field is in a repeater and I saw you fixed a bug related to this in the last release. Presumably the issues are related.  I am on the latest PW dev 3.0.179

 

Uncaught ReferenceError: CKEDITOR is not defined
    at HTMLAnchorElement.<anonymous> (fluency.js:257)

  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...
On 6/5/2021 at 3:54 PM, Mr. NiceGuy said:

Very nice module. It actually also works without a paid account using the free api. The only thing one needs to do is to change one line as the api endpoint is different. 500000 characters for free per month which should probably be enough for many applications. 


classes/DeepL.class.php:    $reqUrl = "https://api-free.deepl.com/v2{$endpoint}?auth_key={$this->apiKey}";

 

I got a javascript error with the module though if I use a non-CK Editor text field (CK editor not found). The textarea field is in a repeater and I saw you fixed a bug related to this in the last release. Presumably the issues are related.  I am on the latest PW dev 3.0.179

 

Uncaught ReferenceError: CKEDITOR is not defined
    at HTMLAnchorElement.<anonymous> (fluency.js:257)

Sorry I've been away for so long. Deciding to buy a house, a puppy, and then remodeling said house while under deadline on deploying a new company API at work is an example of excellent decision making haha.

I didn't know there was a free API and that is awesome news. Might have been introduced since I last worked on the module. I'll be pushing these changes tonight for both the free API option as well as the JS fix. Thank you for your help @Mr. NiceGuy.

@B3ta adding engines is a longer term goal, there are still some refinements to be made before I can do that. Would definitely take a PR if anyone wants to tackle that.

Link to comment
Share on other sites

Fluency now supports the new DeepL free accounts in the latest release (v0.3.0) which also includes some bugfixes. Now live on the repo.

Thank you to @Mr. NiceGuy for notifying me about the new free accounts from DeepL. Very exciting to know that anyone can use Fluency for any project or to play around with at no cost.

I hadn't noticed but since Fluency was released in November 2020 DeepL's supported languages has climbed from 12 to over 25 which is really awesome. Looking forward to seeing what languages they add next. Fluency will always offer all languages that DeepL supports as it pulls this information from the API, so no module updates are ever required to take advantage of these.

Thanks everyone and bugfixes and feature requests are always appreciated over on the Github repo.

  • Like 2
Link to comment
Share on other sites

@FireWire awesome work! 

Currently i'm having some shortcodes [[city]] and seems like i'm getting a strange result https://cln.sh/IztOtO 

Do you think it can be fixed to maintain the [[city]] tag?  Probably it can be solved by adding the glossary option as the desktop app https://cln.sh/tRCCzV

2. for longer text (over 5000 words it seems like the translations is getting stuck on the green screen https://cln.sh/1TObKx

Link to comment
Share on other sites

13 hours ago, B3ta said:

@FireWire awesome work! 

Currently i'm having some shortcodes [[city]] and seems like i'm getting a strange result https://cln.sh/IztOtO 

Do you think it can be fixed to maintain the [[city]] tag?  Probably it can be solved by adding the glossary option as the desktop app https://cln.sh/tRCCzV

2. for longer text (over 5000 words it seems like the translations is getting stuck on the green screen https://cln.sh/1TObKx

That shortcode issue you're seeing was totally my fault. I will push a fix as soon as I can complete it. It will maintain the tags and all other excluded strings properly after that.

5k words! I haven't translated anything that long. As a little test I created a text file with ~5k words in it and the file size was 28kb. The API has a request limit of 30kb and this is almost certainly hitting that ceiling. I need to build out a way for Fluency to analyze the size of the translation, split it up into multiple requests to DeepL, and the re-assemble the multiple translations back into one. Unfortunately this is a hard limitation by the DeepL API so the workaround will have to come from me. I'll put that on the to-do list.

  • Thanks 1
Link to comment
Share on other sites

40 minutes ago, B3ta said:

Thank you for your quick reply and fixes! 
I just sent you a paypal donation. 

Many thanks! Let me know if anything else pops up.

Link to comment
Share on other sites

On 6/23/2021 at 4:42 AM, FireWire said:

Many thanks! Let me know if anything else pops up.

It seems like the the issue is popping up even with smaller text form, around 800 words even on the pro version of the API. 

It would be nice if you can take a look and see if it can get fixed.

 

Thanks!

Link to comment
Share on other sites

On 6/24/2021 at 9:00 AM, B3ta said:

It seems like the the issue is popping up even with smaller text form, around 800 words even on the pro version of the API. 

It would be nice if you can take a look and see if it can get fixed.

 

Thanks!

I've seen translations take varying times depending on length and language. Are you letting it work for a while?  I will test more with longer content and look into how DeepL handles it.

  • Like 1
Link to comment
Share on other sites

@FireWire Do you think you can bring an update to module to translate from secondary language to main language?

Eg:  Site Main lang is DE and sec lang is EN , but the article writer writes the text in english and the client would be able to translate from EN to DE (main language) 

 

  • Like 1
Link to comment
Share on other sites

6 hours ago, B3ta said:

@FireWire Do you think you can bring an update to module to translate from secondary language to main language?

Eg:  Site Main lang is DE and sec lang is EN , but the article writer writes the text in english and the client would be able to translate from EN to DE (main language) 

 

I thought about building that out from the start but wanted to focus on getting the core of the module working first in a way that matched the way ProcessWire's multi-language setup works where there is a primary language and secondary languages.

Including the general translation tool in the top menubar where any text can be translated from/to any language was a short term solution for that limitation. Once Fluency is able to mature and all of the bugs fleshed out then revisiting that would be an option.

  • Like 1
Link to comment
Share on other sites

@B3ta Update on the errors with lengthy content. Unfortunately it's not an issue with the module from what I can see. I was able to find out where the limitation on length was down to the sentence.

When the API is used to translate it returns standardized responses that returns the translated content when it is successful, or errors that give instructions on what went wrong. Unfortunately the issue we are experiencing causes the DeepL API to return nothing (null) which means that their system experiences and issue but does not respond the way that they say it should.

I'm going to be sending in a support request to DeepL detailing the issue and see what they say. Will report back with updates but as of right now it looks like Fluency can only handle medium amounts of text.

Link to comment
Share on other sites

22 hours ago, FireWire said:

@B3ta Update on the errors with lengthy content. Unfortunately it's not an issue with the module from what I can see. I was able to find out where the limitation on length was down to the sentence.

When the API is used to translate it returns standardized responses that returns the translated content when it is successful, or errors that give instructions on what went wrong. Unfortunately the issue we are experiencing causes the DeepL API to return nothing (null) which means that their system experiences and issue but does not respond the way that they say it should.

I'm going to be sending in a support request to DeepL detailing the issue and see what they say. Will report back with updates but as of right now it looks like Fluency can only handle medium amounts of text.

@B3ta


That was all totally wrong. I was improperly making the DeepL API call with the wrong method which was totally my fault.

After working on the issue I've tested translations with a lot of content and have had success up to ~22,000 words (rich text in a CKEditor field). If your PW fields has more than that, I'm impressed haha. That will have to be the new ceiling and I'll engineer a solution if people start hitting that. When working with content up to those levels though I think DeepL may be being generous with their stated API request size limit so keep in mind that number could fluctuate- just hypothesizing though. You should be more than okay when translating 5000 words though.

Will push a new version soon and let you know when it is ready.

Link to comment
Share on other sites

New version released. Fixed issue when translating larger volumes of text. In testing I was successful in translating over 20,000+ words in one field. When you get to large volumes of content like that there is a very noticeable amount of time it takes to complete that request, but it does it.

Download the newest alpha here: Fluency 0.3.2

This should solve your issues @B3ta, let me know if you experience any issues.

  • Thanks 2
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Robin S
      This module lets you add some custom menu items to the main admin menu, and you can set the dropdown links dynamically in a hook if needed.
      Sidenote: the module config uses some repeatable/sortable rows for the child link settings, similar to the ProFields Table interface. The data gets saved as JSON in a hidden textarea field. Might be interesting to other module developers?
      Custom Admin Menus
      Adds up to three custom menu items with optional dropdowns to the main admin menu.
      The menu items can link to admin pages, front-end pages, or pages on external websites.
      The links can be set to open in a new browser tab, and child links in the dropdown can be given an icon.
      Requires ProcessWire v3.0.178 or newer.
      Screenshots
      Example of menu items

      Module config for the menus

      Link list shown when parent menu item is not given a URL

      Advanced
      Setting child menu items dynamically
      If needed you can set the child menu items dynamically using a hook.
      Example:
      $wire->addHookAfter('CustomAdminMenus::getMenuChildren', function(HookEvent $event) { // The menu number is the first argument $menu_number = $event->arguments(0); if($menu_number === 1) { $colours = $event->wire()->pages->findRaw('template=colour', ['title', 'url', 'page_icon']); $children = []; foreach($colours as $colour) { // Each child item should be an array with the following keys $children[] = [ 'icon' => $colour['page_icon'], 'label' => $colour['title'], 'url' => $colour['url'], 'newtab' => false, ]; } $event->return = $children; } }); Create multiple levels of flyout menus
      It's also possible to create multiple levels of flyout submenus using a hook.

      For each level a submenu can be defined in a "children" item. Example:
      $wire->addHookAfter('CustomAdminMenus::getMenuChildren', function(HookEvent $event) { // The menu number is the first argument $menu_number = $event->arguments(0); if($menu_number === 1) { $children = [ [ 'icon' => 'adjust', 'label' => 'One', 'url' => '/one/', 'newtab' => false, ], [ 'icon' => 'anchor', 'label' => 'Two', 'url' => '/two/', 'newtab' => false, 'children' => [ [ 'icon' => 'child', 'label' => 'Red', 'url' => '/red/', 'newtab' => false, ], [ 'icon' => 'bullhorn', 'label' => 'Green', 'url' => '/green/', 'newtab' => false, 'children' => [ [ 'icon' => 'wifi', 'label' => 'Small', 'url' => '/small/', 'newtab' => true, ], [ 'icon' => 'codepen', 'label' => 'Medium', 'url' => '/medium/', 'newtab' => false, ], [ 'icon' => 'cogs', 'label' => 'Large', 'url' => '/large/', 'newtab' => false, ], ] ], [ 'icon' => 'futbol-o', 'label' => 'Blue', 'url' => '/blue/', 'newtab' => true, ], ] ], [ 'icon' => 'hand-o-left', 'label' => 'Three', 'url' => '/three/', 'newtab' => false, ], ]; $event->return = $children; } }); Showing/hiding menus according to user role
      You can determine which menu items can be seen by a role by checking the user's role in the hook.
      For example, if a user has or lacks a role you could include different child menu items in the hook return value. Or if you want to conditionally hide a custom menu altogether you can set the return value to false. Example:
      $wire->addHookAfter('CustomAdminMenus::getMenuChildren', function(HookEvent $event) { // The menu number is the first argument $menu_number = $event->arguments(0); $user = $event->wire()->user; // For custom menu number 1... if($menu_number === 1) { // ...if user does not have some particular role... if(!$user->hasRole('foo')) { // ...do not show the menu $event->return = false; } } });  
      https://github.com/Toutouwai/CustomAdminMenus
      https://processwire.com/modules/custom-admin-menus/
    • By tcnet
      This module for ProcessWire sends a notification email for each failed login attempt. Similar modules exists already in the module directory of ProcessWire. However, this module is designed to notify, even if specified user doesn't exist.
      Settings
      The settings for this module are located in the menu Modules=>Configure=>LoginFailNotifier.
      Notification email
      Specifies the email address to which the notification emails should be sent.
        Email subject
      Specifies the subject line for the notification email.
        Post variables
      Specifies the $_POST variables to be included in the notification email. Each variable must be separated by a comma. For example: login_name,login_pass
        Server variables
      Specifies the $_SERVER variables to be included in the notification email. Each variable must be separated by a comma. For example: REMOTE_ADDR,HTTP_USER_AGENT
      Link to ProcessWire module directory:
      https://processwire.com/modules/login-fail-notifier/
      Link to github.com:
      https://github.com/techcnet/LoginFailNotifier
    • By Fokke
      ProcessWire 3.x markup module for rendering meta tags in HTML document head section. Note that this module is not a full-blown SEO solution, but rather a simple tool for rendering meta tags based on module configuration. Adding custom meta tags is also supported.
      Built-in meta tags
      The following meta tags are supported out-of-the-box:
      Document title consisting of page title and site name Character set Canonical Viewport Description Keywords Hreflang tags Open Graph og:title og:site_name og:type og:url og:description og:image og:image:width og:image:height Twitter meta tags twitter:card twitter:site twitter:creator twitter:title twitter:description twitter:image Facebook meta tags fb:app_id The full documentation with configurable options can be found here: https://github.com/Fokke-/MarkupMetadata
       
      Requirements:
      ProcessWire>=3.0.0 PHP >=7.1 Installation using Composer
      composer require fokke/markup-metadata Manual installation
      Download latest version from https://github.com/Fokke-/MarkupMetadata/archive/master.zip Extract module files to site/modules/MarkupMetadata directory.
    • By m.sieber
      ITRK-Service for ProcessWire
      Module for the automated transfer of imprint, data protection declaration and terms and conditions from IT-Recht Kanzlei to your ProcessWire installation
      What is ITRK Service for ProcessWire?
      ITRK-Service for ProcessWire is a free module for ProcessWire CMS. It provides an interface to the update service of IT-Recht Kanzlei, via which the legal texts of your online presence are automatically updated. In this way, the texts remain legally secure and warning-proof in the long term. Imprint, data protection declaration, revocation and general terms and conditions are currently supported.
      You can find our documentation (in german language) here: https://www.pupit.de/itrk-service-for-processwire/dokumentation/

      Download: https://www.pupit.de/itrk-service-for-processwire/
      Github: https://github.com/pupit-de/pwItrkServiceConnector
    • By LuisM
      Symprowire is a PHP MVC Framework based and built on Symfony using ProcessWire 3.x as DBAL and Service-Provider
      It acts as a Drop-In Replacement Module to handle the Request/Response outside the ProcessWire Admin. Even tough Symfony or any other mature MVC Framework could be intimidating at first, Symprowire tries to abstract Configuration and Symfony Internals away as much as possible to give you a quick start and lift the heavy work for you.
      The main Goal is to give an easy path to follow an MVC Approach during development with ProcessWire and open up the available eco-system.
      You can find the GitHub Repo and more Information here: https://github.com/Luis85/symprowire
      Documentation
      The Symprowire Wiki https://github.com/Luis85/symprowire/wiki How to create a simple Blog with Symprowire https://github.com/Luis85/symprowire/wiki/Symprowire-Blog-Tutorial Last Update
      16.07.2021 // RC 1 v0.6.0 centralized ProcessWire access trough out the Application by wrapping to a Service https://github.com/Luis85/symprowire/releases/tag/v0.6.0-rc-1 Requirements
      PHP ^7.4 Fresh ProcessWire ^3.0.181 with a Blank Profile Composer 2 (v1 should work, not recommended) The usual Symfony Requirements Features
      Twig Dependency Injection Monolog for Symprowire Support for .env YAML Configuration Symfony Console and Console Commands Symfony Webprofiler Full ProcessWire access inside your Controller and Services Webpack Encore support Caveats
      Symfony is no small Framework and will come with a price in terms of Memory Usage and added Overhead. To give you a taste I installed Tracy Debugger alongside to compare ProcessWire profiling with the included Symfony Webprofiler

      So in a fresh install Symprowire would atleast add another 2MB of Memory usage and around 40ms in response time, should be less in production due to the added overhead of the Webprofiler in dev env
       
×
×
  • Create New...