Jump to content

Fluency - Integrated DeepL Powered Content Translation


FireWire

Recommended Posts

19 hours ago, adrian said:

@FireWire - just a reminder to remove the bd($content) call in Fluency.module.php 

Also, just want to agree with @xportde's request for a dedicated permission. I have already changed it locally to add a "fluency" permission. In my use case I actually needed to give the guest user this permission so that I could do translations via a cron called script.

Thanks again for your work on this - I am now using it in production.

Pushed the fix that removed the bd() call. Thank you for the reminder. I put some work in over the weekend on the permission feature. The new permission is named "fluency-translate" in case you want to rename yours to match the later releases. This release also formally adds the 5th Fluency->translate() argument to feed in additional API parameters.

Everything is always a time crunch between holidays here, looking forward to when I can get back to working on more changes. Permission will be the next feature.

Excellent to hear that it's in production! We've been using it and it's been smooth sailing.

  • Like 1
Link to comment
Share on other sites

@FireWire - not sure if you want to bother with this, but I just found out that DeepL doesn't yet support translating between US and UK English. I was hoping to be able to change things like neighbor to neighbour. I have contacted DeepL and asked them to consider supporting this. Strange thing is that you can actually translate from US English to Spanish to UK English, eg: neighbor -> vecino -> neighbour.

Hopefully they'll support direct translation in the future, but until then I wonder if you want to remove that option from the Translation process module?

  • Like 1
Link to comment
Share on other sites

On 12/9/2020 at 7:49 PM, adrian said:

@FireWire - not sure if you want to bother with this, but I just found out that DeepL doesn't yet support translating between US and UK English. I was hoping to be able to change things like neighbor to neighbour. I have contacted DeepL and asked them to consider supporting this. Strange thing is that you can actually translate from US English to Spanish to UK English, eg: neighbor -> vecino -> neighbour.

Hopefully they'll support direct translation in the future, but until then I wonder if you want to remove that option from the Translation process module?

That's interesting, I'd never actually tested that. I'll make a note to disable those translation connections in the module.

Also really odd that capability doesn't exist... seems like literally the easiest translation they'd have to do haha.

  • Like 1
Link to comment
Share on other sites

Important update to the module with a new release. Recommended that all users upgrade to the latest version

The module has been updated with critical fixes and feature requests fulfilled.

  • The module has been updated to fix an issue with lengthy content that would result in failure, mostly important for CKEditor fields that include markup in the content. This was due to the admin AJAX call to the module using a GET request where a POST request should have been used to prevent running into URL length issues.
  • Translation was pulling from the wrong location in CKEditors so it was translating the last saved content, not the latest changes made in the field before page save.
  • Use of Fluency is now permission based as requested. After upgrading all roles who should have access must have the fluency-translate permission added
  • The configuration screen now has a new setting for the Preserve Formatting feature which addresses the issue reported in this thread where short strings or phrases were being returned with additional/missing punctuation. (internally this uses the preserve_formatting DeepL API parameter). It is recommended that this be enabled unless you have a specific reason not to use it.
  • Public methods have been added to the Fluency module so that translation, usage stats, and available languages can be pulled from calling the Fluency module or by using the DeepL class directly. README has been updated with documentation for usage.
  • A new parameter has been added to both the Fluency and DeepL translate methods that allow you to pass in any DeepL API key/values with your request. If you are currently calling either the module or DeepL class directly, please read the documentation as the parameters have changed.
  • Return values from the Fluency module now match the return values from the DeepL module for consistency. This includes passing DeepL's HTTP response for use in programmatic conditions.
  • The module has been moved from Gitlab to Github to adhere to ProcessWire's module directory requirements for future listing.

As usual, more to follow! Latest: https://github.com/SkyLundy/Fluency-Translation/archive/main.zip

Paging @adrian & @xportde & @bartelsmedia as you've been Fluency's alpha testers extraordinaire.

  • Like 2
Link to comment
Share on other sites

  • 1 month later...

@FireWire - sorry for the silence on your last post - I haven't had a chance to test this new version yet - hopefully soon.

A question for you thought - have you looked at: https://lingvanex.com/translationapi/ - they are much cheaper than DeepL and support a LOT more languages. This post is also worth a read: https://lingvanex.com/history-lingvanex/

Anyway, I am need of Tagalog translation and am thinking about Lingvanex but would like to know if you investigated it and decided against it for some reason.

Thanks for your input.

Link to comment
Share on other sites

34 minutes ago, adrian said:

@FireWire - sorry for the silence on your last post - I haven't had a chance to test this new version yet - hopefully soon.

A question for you thought - have you looked at: https://lingvanex.com/translationapi/ - they are much cheaper than DeepL and support a LOT more languages. This post is also worth a read: https://lingvanex.com/history-lingvanex/

Anyway, I am need of Tagalog translation and am thinking about Lingvanex but would like to know if you investigated it and decided against it for some reason.

Thanks for your input.

Really no problem at all. I've released a newer version since then that addresses an issue that I didn't see come up in testing previously. Just a courtesy tag in case the bugfixes end up helping someone.

I haven't heard of Lingvanex before so I didn't know to look into them. Initially when I found DeepL I was really impressed by the service but needed someone else to really verify what was going on because I only speak English. I had a coworker who speaks Spanish look at the results and he was very impressed. I kind of stopped there and didn't think to look further. Overall if someone was able to verify the quality of translation of any translation then that's all that matters.

I don't have any personal allegiance to DeepL, it's just what I found and what I heard about on the PW forums which also nudged me towards them. It would be interesting in the future to update Fluency to allow the dev to select a translation engine, and the way the module is built I think it could be done well. I wish I could work on that now but I just had 3 clients contact me for work and that's in addition to my day job...

 

  • Like 2
Link to comment
Share on other sites

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

  • Like 2
Link to comment
Share on other sites

16 hours ago, adrian said:

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

Appreciate the consideration! I tried to make Fluency as modular as possible so hopefully it shouldn't be too hard to add another engine. If you take a look at how DeepL works with Fluency, it exists as it's own class and Fluency doesn't know how or what happens in DeepL, and vice versa. Following that structure, you could create a Lingvanex class that Fluency sends data to. Matching DeepL method parameters when calling it and data structures it returns could make it pretty straightforward.

An idea I had would be to create a bootTranslationEngine() method that is called in ready() which would determine what engine/API key is available and instantiate the proper translator class in something like $this->translationEngine. That would replace $this->deepL in the Fluency module. I would make it a goal to abstract out any references to any brand of translator in the Fluency module's logic.

As long as getClientBootData() and translate() take and return the same structure of data you wouldn't even need to touch the front end.

I haven't really written any code that is collaborated on before so I hope that the commenting and overall structure makes sense. If you ever have any questions or want a shortcut to figuring something I did out then message me and we can chat.

 

 

  • Like 2
Link to comment
Share on other sites

  • 1 month later...
On 2/19/2021 at 7:22 AM, adrian said:

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

HI @adrian, any news on Lingvanex PR? 

Link to comment
Share on other sites

  • 4 weeks later...
On 4/14/2021 at 10:02 AM, B3ta said:

HI @adrian, any news on Lingvanex PR? 

Sorry, because I don't actually use a translation service to translate content in the admin I took the easy way out and just implemented a direct curl request to the Lingvanex API.

Link to comment
Share on other sites

  • 4 weeks later...

Very nice module. It actually also works without a paid account using the free api. The only thing one needs to do is to change one line as the api endpoint is different. 500000 characters for free per month which should probably be enough for many applications. 

classes/DeepL.class.php:    $reqUrl = "https://api-free.deepl.com/v2{$endpoint}?auth_key={$this->apiKey}";

 

I got a javascript error with the module though if I use a non-CK Editor text field (CK editor not found). The textarea field is in a repeater and I saw you fixed a bug related to this in the last release. Presumably the issues are related.  I am on the latest PW dev 3.0.179

 

Uncaught ReferenceError: CKEDITOR is not defined
    at HTMLAnchorElement.<anonymous> (fluency.js:257)

  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...
On 6/5/2021 at 3:54 PM, Mr. NiceGuy said:

Very nice module. It actually also works without a paid account using the free api. The only thing one needs to do is to change one line as the api endpoint is different. 500000 characters for free per month which should probably be enough for many applications. 


classes/DeepL.class.php:    $reqUrl = "https://api-free.deepl.com/v2{$endpoint}?auth_key={$this->apiKey}";

 

I got a javascript error with the module though if I use a non-CK Editor text field (CK editor not found). The textarea field is in a repeater and I saw you fixed a bug related to this in the last release. Presumably the issues are related.  I am on the latest PW dev 3.0.179

 

Uncaught ReferenceError: CKEDITOR is not defined
    at HTMLAnchorElement.<anonymous> (fluency.js:257)

Sorry I've been away for so long. Deciding to buy a house, a puppy, and then remodeling said house while under deadline on deploying a new company API at work is an example of excellent decision making haha.

I didn't know there was a free API and that is awesome news. Might have been introduced since I last worked on the module. I'll be pushing these changes tonight for both the free API option as well as the JS fix. Thank you for your help @Mr. NiceGuy.

@B3ta adding engines is a longer term goal, there are still some refinements to be made before I can do that. Would definitely take a PR if anyone wants to tackle that.

Link to comment
Share on other sites

Fluency now supports the new DeepL free accounts in the latest release (v0.3.0) which also includes some bugfixes. Now live on the repo.

Thank you to @Mr. NiceGuy for notifying me about the new free accounts from DeepL. Very exciting to know that anyone can use Fluency for any project or to play around with at no cost.

I hadn't noticed but since Fluency was released in November 2020 DeepL's supported languages has climbed from 12 to over 25 which is really awesome. Looking forward to seeing what languages they add next. Fluency will always offer all languages that DeepL supports as it pulls this information from the API, so no module updates are ever required to take advantage of these.

Thanks everyone and bugfixes and feature requests are always appreciated over on the Github repo.

  • Like 2
Link to comment
Share on other sites

@FireWire awesome work! 

Currently i'm having some shortcodes [[city]] and seems like i'm getting a strange result https://cln.sh/IztOtO 

Do you think it can be fixed to maintain the [[city]] tag?  Probably it can be solved by adding the glossary option as the desktop app https://cln.sh/tRCCzV

2. for longer text (over 5000 words it seems like the translations is getting stuck on the green screen https://cln.sh/1TObKx

Link to comment
Share on other sites

13 hours ago, B3ta said:

@FireWire awesome work! 

Currently i'm having some shortcodes [[city]] and seems like i'm getting a strange result https://cln.sh/IztOtO 

Do you think it can be fixed to maintain the [[city]] tag?  Probably it can be solved by adding the glossary option as the desktop app https://cln.sh/tRCCzV

2. for longer text (over 5000 words it seems like the translations is getting stuck on the green screen https://cln.sh/1TObKx

That shortcode issue you're seeing was totally my fault. I will push a fix as soon as I can complete it. It will maintain the tags and all other excluded strings properly after that.

5k words! I haven't translated anything that long. As a little test I created a text file with ~5k words in it and the file size was 28kb. The API has a request limit of 30kb and this is almost certainly hitting that ceiling. I need to build out a way for Fluency to analyze the size of the translation, split it up into multiple requests to DeepL, and the re-assemble the multiple translations back into one. Unfortunately this is a hard limitation by the DeepL API so the workaround will have to come from me. I'll put that on the to-do list.

Link to comment
Share on other sites

40 minutes ago, B3ta said:

Thank you for your quick reply and fixes! 
I just sent you a paypal donation. 

Many thanks! Let me know if anything else pops up.

Link to comment
Share on other sites

On 6/23/2021 at 4:42 AM, FireWire said:

Many thanks! Let me know if anything else pops up.

It seems like the the issue is popping up even with smaller text form, around 800 words even on the pro version of the API. 

It would be nice if you can take a look and see if it can get fixed.

 

Thanks!

Link to comment
Share on other sites

On 6/24/2021 at 9:00 AM, B3ta said:

It seems like the the issue is popping up even with smaller text form, around 800 words even on the pro version of the API. 

It would be nice if you can take a look and see if it can get fixed.

 

Thanks!

I've seen translations take varying times depending on length and language. Are you letting it work for a while?  I will test more with longer content and look into how DeepL handles it.

  • Like 1
Link to comment
Share on other sites

@FireWire Do you think you can bring an update to module to translate from secondary language to main language?

Eg:  Site Main lang is DE and sec lang is EN , but the article writer writes the text in english and the client would be able to translate from EN to DE (main language) 

 

  • Like 1
Link to comment
Share on other sites

6 hours ago, B3ta said:

@FireWire Do you think you can bring an update to module to translate from secondary language to main language?

Eg:  Site Main lang is DE and sec lang is EN , but the article writer writes the text in english and the client would be able to translate from EN to DE (main language) 

 

I thought about building that out from the start but wanted to focus on getting the core of the module working first in a way that matched the way ProcessWire's multi-language setup works where there is a primary language and secondary languages.

Including the general translation tool in the top menubar where any text can be translated from/to any language was a short term solution for that limitation. Once Fluency is able to mature and all of the bugs fleshed out then revisiting that would be an option.

  • Like 1
Link to comment
Share on other sites

@B3ta Update on the errors with lengthy content. Unfortunately it's not an issue with the module from what I can see. I was able to find out where the limitation on length was down to the sentence.

When the API is used to translate it returns standardized responses that returns the translated content when it is successful, or errors that give instructions on what went wrong. Unfortunately the issue we are experiencing causes the DeepL API to return nothing (null) which means that their system experiences and issue but does not respond the way that they say it should.

I'm going to be sending in a support request to DeepL detailing the issue and see what they say. Will report back with updates but as of right now it looks like Fluency can only handle medium amounts of text.

Link to comment
Share on other sites

22 hours ago, FireWire said:

@B3ta Update on the errors with lengthy content. Unfortunately it's not an issue with the module from what I can see. I was able to find out where the limitation on length was down to the sentence.

When the API is used to translate it returns standardized responses that returns the translated content when it is successful, or errors that give instructions on what went wrong. Unfortunately the issue we are experiencing causes the DeepL API to return nothing (null) which means that their system experiences and issue but does not respond the way that they say it should.

I'm going to be sending in a support request to DeepL detailing the issue and see what they say. Will report back with updates but as of right now it looks like Fluency can only handle medium amounts of text.

@B3ta


That was all totally wrong. I was improperly making the DeepL API call with the wrong method which was totally my fault.

After working on the issue I've tested translations with a lot of content and have had success up to ~22,000 words (rich text in a CKEditor field). If your PW fields has more than that, I'm impressed haha. That will have to be the new ceiling and I'll engineer a solution if people start hitting that. When working with content up to those levels though I think DeepL may be being generous with their stated API request size limit so keep in mind that number could fluctuate- just hypothesizing though. You should be more than okay when translating 5000 words though.

Will push a new version soon and let you know when it is ready.

Link to comment
Share on other sites

New version released. Fixed issue when translating larger volumes of text. In testing I was successful in translating over 20,000+ words in one field. When you get to large volumes of content like that there is a very noticeable amount of time it takes to complete that request, but it does it.

Download the newest alpha here: Fluency 0.3.2

This should solve your issues @B3ta, let me know if you experience any issues.

  • Thanks 2
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By LuisM
      Symprowire is a PHP MVC Framework based and built on Symfony using ProcessWire 3.x as DBAL and Service-Provider
      It acts as a Drop-In Replacement Module to handle the Request/Response outside the ProcessWire Admin. Even tough Symfony or any other mature MVC Framework could be intimidating at first, Symprowire tries to abstract Configuration and Symfony Internals away as much as possible to give you a quick start and lift the heavy work for you.
      The main Goal is to give an easy path to follow an MVC Approach during development with ProcessWire and open up the available eco-system.
      You can find the GitHub Repo and more Information here: https://github.com/Luis85/symprowire
      Documentation
      The Symprowire Wiki https://github.com/Luis85/symprowire/wiki How to create a simple Blog with Symprowire https://github.com/Luis85/symprowire/wiki/Symprowire-Blog-Tutorial Last Update
      16.07.2021 // RC 1 v0.6.0 centralized ProcessWire access trough out the Application by wrapping to a Service https://github.com/Luis85/symprowire/releases/tag/v0.6.0-rc-1 Requirements
      PHP ^7.4 Fresh ProcessWire ^3.0.181 with a Blank Profile Composer 2 (v1 should work, not recommended) The usual Symfony Requirements Features
      Twig Dependency Injection Monolog for Symprowire Support for .env YAML Configuration Symfony Console and Console Commands Symfony Webprofiler Full ProcessWire access inside your Controller and Services Webpack Encore support Caveats
      Symfony is no small Framework and will come with a price in terms of Memory Usage and added Overhead. To give you a taste I installed Tracy Debugger alongside to compare ProcessWire profiling with the included Symfony Webprofiler

      So in a fresh install Symprowire would atleast add another 2MB of Memory usage and around 40ms in response time, should be less in production due to the added overhead of the Webprofiler in dev env
       
    • By tcnet
      PageViewStatistic for ProcessWire is a module to log page visits of the CMS. The records including some basic information like IP-address, browser, operating system, requested page and originate page. Please note that this module doesn't claim to be the best or most accurate.
      Advantages
      One of the biggest advantage is that this module doesn't require any external service like Google Analytics or similar. You don't have to modify your templates either. There is also no Javascript or image required.
      Disadvantages
      There is only one disadvantage. This module doesn't record visits if the browser loads the page from its browser cache. To prevent the browser from loading the page from its cache, add the following meta tags to the header of your page:
      <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" /> <meta http-equiv="Pragma" content="no-cache" /> <meta http-equiv="Expires" content="0" /> How to use
      The records can be accessed via the Setup-menu of the CMS backend. The first dropdown control changes the view mode.

      Detailed records
      View mode "Detailed records" shows all visits of the selected day individually with IP-address, browser, operating system, requested page and originate page. Click the update button to see new added records.

      Cached visitor records
      View modes other than "Detailed records" are cached visitor counts which will be collected on a daily basis from the detailed records. This procedure ensures a faster display even with a large number of data records. Another advantage is that the detailed records can be deleted while the cache remains. The cache can be updated manually or automatically in a specified time period. Multiple visits from the same IP address on the same day are counted as a single visitor.

      Upgrade from older versions
      Cached visitor counts is new in version 1.0.8. If you just upgraded from an older version you might expire a delay or even an error 500 if you display cached visitor counts. The reason for this is that the cache has to be created from the records. This can take longer if your database contains many records. Sometimes it might hit the maximally execution time. Don't worry about that and keep reloading the page until the cache is completely created.
      Special Feature
      PageViewStatistic for ProcessWire can record the time a visitor viewed the page. This feature is deactivated by default. To activate open the module configuration page and activate "Record view time". If activated you will find a new column "S." in the records which means the time of view in seconds. With every page request, a Javascript code is inserted directly after the <body> tag. Every time the visitor switches to another tab or closes the tab, this script reports the number of seconds the tab was visible. The initial page request is recorded only as a hyphen (-).

      New in version 1.1.0
      A new feature comes with version 1.1.0 which offers to record user names of loggedin visitors. Just activate "Record user names" and "Record loggedin user" in the module settings.
      Settings
      You can access the module settings by clicking the Configuration button at the bottom of the records page. The settings page is also available in the menu: Modules->Configure->ProcessPageViewStat.
      IP2Location
      This module uses the IP2Location database from: http://www.ip2location.com. This database is required to obtain the country from the IP address. IP2Location updates this database at the begin of every month. The settings of ProcessPageViewStat offers the ability to automatically download the database monthly. Please note, that automatically download will not work if your webspace doesn't allow allow_url_fopen.
      Dragscroll
      This module uses DragScroll. A JavaScript available from: http://github.com/asvd/dragscroll. Dragscroll adds the ability in view mode "Day" to drag the records horizontally with the mouse pointer.
      parseUserAgentStringClass
      This module uses the PHP class parseUserAgentStringClass available from: http://www.toms-world.org/blog/parseuseragentstring/. This class is required to filter out the browser type and operating system from the server request.
    • By clsource
      Inertia Adapter ProcessWire Module
      Hello! Long time no see.
      I created this module so you can use Inertia.js (https://inertiajs.com/) with ProcessWire.
      Description
      Inertia allows you to create fully client-side rendered, single-page apps, without much of the complexity that comes with modern SPAs. It does this by leveraging existing server-side frameworks.
      Inertia isn’t a framework, nor is it a replacement to your existing server-side or client-side frameworks. Rather, it’s designed to work with them. Think of Inertia as glue that connects the two. Inertia comes with three official client-side adapters (React, Vue, and Svelte).
      This is an adapter for ProcessWire. Inertia replaces PHP views altogether by returning JavaScript components from controller actions. Those components can be built with your frontend framework of choice.
      Links
      - https://github.com/joyofpw/inertia
      - https://github.com/joyofpw/inertia-svelte-mix-pw
      - https://inertiajs.com/
      Screenshots


       
    • By sms77io
      Hi all,
      we made a small module for sending SMS via Sms77.io. It supports sending to one and multiple users.
      You can download it from GitHub and follow the instructions on how to install it - it is quite easy. An API key is required for sending, get yours for free @ Sms77 and receive 0,50 €.
      Hope this helps somebody and we are open for improvement suggestions!
       
      Best regards
      André
    • By tcnet
      This module implements the website live chat service from tawk.to. Actually the module doesn't have to do much. It just need to inserted a few lines of JavaScript just before the closing body tag </body> on each side. However, the module offers additional options to display the widget only on certain pages.
      Create an account
      Visit https://www.tawk.to and create an account. It's free! At some point you will reach a page where you can copy the required JavaScript-code.

      Open the module settings and paste the JavaScript-code into the field as shown below. Click "Submit" and that's all.

      Open the module settings
      The settings for this module are located int the menu Modules=>Configure=>LiveChatTawkTo.

       
×
×
  • Create New...