module Fluency - The complete translation enhancement suite for ProcessWire

[FireWire]

19 hours ago, adrian said:

@FireWire - just a reminder to remove the bd($content) call in Fluency.module.php 

Also, just want to agree with @xportde's request for a dedicated permission. I have already changed it locally to add a "fluency" permission. In my use case I actually needed to give the guest user this permission so that I could do translations via a cron called script.

Thanks again for your work on this - I am now using it in production.

Pushed the fix that removed the bd() call. Thank you for the reminder. I put some work in over the weekend on the permission feature. The new permission is named "fluency-translate" in case you want to rename yours to match the later releases. This release also formally adds the 5th Fluency->translate() argument to feed in additional API parameters.

Everything is always a time crunch between holidays here, looking forward to when I can get back to working on more changes. Permission will be the next feature.

Excellent to hear that it's in production! We've been using it and it's been smooth sailing.

[adrian]

@FireWire - not sure if you want to bother with this, but I just found out that DeepL doesn't yet support translating between US and UK English. I was hoping to be able to change things like neighbor to neighbour. I have contacted DeepL and asked them to consider supporting this. Strange thing is that you can actually translate from US English to Spanish to UK English, eg: neighbor -> vecino -> neighbour.

Hopefully they'll support direct translation in the future, but until then I wonder if you want to remove that option from the Translation process module?

[FireWire]

On 12/9/2020 at 7:49 PM, adrian said:

@FireWire - not sure if you want to bother with this, but I just found out that DeepL doesn't yet support translating between US and UK English. I was hoping to be able to change things like neighbor to neighbour. I have contacted DeepL and asked them to consider supporting this. Strange thing is that you can actually translate from US English to Spanish to UK English, eg: neighbor -> vecino -> neighbour.

Hopefully they'll support direct translation in the future, but until then I wonder if you want to remove that option from the Translation process module?

That's interesting, I'd never actually tested that. I'll make a note to disable those translation connections in the module.

Also really odd that capability doesn't exist... seems like literally the easiest translation they'd have to do haha.

[FireWire]

Important update to the module with a new release. Recommended that all users upgrade to the latest version

The module has been updated with critical fixes and feature requests fulfilled.

• The module has been updated to fix an issue with lengthy content that would result in failure, mostly important for CKEditor fields that include markup in the content. This was due to the admin AJAX call to the module using a GET request where a POST request should have been used to prevent running into URL length issues.
• Translation was pulling from the wrong location in CKEditors so it was translating the last saved content, not the latest changes made in the field before page save.
• Use of Fluency is now permission based as requested. After upgrading all roles who should have access must have the fluency-translate permission added
• The configuration screen now has a new setting for the Preserve Formatting feature which addresses the issue reported in this thread where short strings or phrases were being returned with additional/missing punctuation. (internally this uses the preserve_formatting DeepL API parameter). It is recommended that this be enabled unless you have a specific reason not to use it.
• Public methods have been added to the Fluency module so that translation, usage stats, and available languages can be pulled from calling the Fluency module or by using the DeepL class directly. README has been updated with documentation for usage.
• A new parameter has been added to both the Fluency and DeepL translate methods that allow you to pass in any DeepL API key/values with your request. If you are currently calling either the module or DeepL class directly, please read the documentation as the parameters have changed.
• Return values from the Fluency module now match the return values from the DeepL module for consistency. This includes passing DeepL's HTTP response for use in programmatic conditions.
• The module has been moved from Gitlab to Github to adhere to ProcessWire's module directory requirements for future listing.

As usual, more to follow! Latest: https://github.com/SkyLundy/Fluency-Translation/archive/main.zip

Paging @adrian & @xportde & @bartelsmedia as you've been Fluency's alpha testers extraordinaire.

[adrian]

@FireWire - sorry for the silence on your last post - I haven't had a chance to test this new version yet - hopefully soon.

A question for you thought - have you looked at: https://lingvanex.com/translationapi/ - they are much cheaper than DeepL and support a LOT more languages. This post is also worth a read: https://lingvanex.com/history-lingvanex/

Anyway, I am need of Tagalog translation and am thinking about Lingvanex but would like to know if you investigated it and decided against it for some reason.

Thanks for your input.

[FireWire]

34 minutes ago, adrian said:

@FireWire - sorry for the silence on your last post - I haven't had a chance to test this new version yet - hopefully soon.

A question for you thought - have you looked at: https://lingvanex.com/translationapi/ - they are much cheaper than DeepL and support a LOT more languages. This post is also worth a read: https://lingvanex.com/history-lingvanex/

Anyway, I am need of Tagalog translation and am thinking about Lingvanex but would like to know if you investigated it and decided against it for some reason.

Thanks for your input.

Really no problem at all. I've released a newer version since then that addresses an issue that I didn't see come up in testing previously. Just a courtesy tag in case the bugfixes end up helping someone.

I haven't heard of Lingvanex before so I didn't know to look into them. Initially when I found DeepL I was really impressed by the service but needed someone else to really verify what was going on because I only speak English. I had a coworker who speaks Spanish look at the results and he was very impressed. I kind of stopped there and didn't think to look further. Overall if someone was able to verify the quality of translation of any translation then that's all that matters.

I don't have any personal allegiance to DeepL, it's just what I found and what I heard about on the PW forums which also nudged me towards them. It would be interesting in the future to update Fluency to allow the dev to select a translation engine, and the way the module is built I think it could be done well. I wish I could work on that now but I just had 3 clients contact me for work and that's in addition to my day job...

 

[adrian]

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

[FireWire]

16 hours ago, adrian said:

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

Appreciate the consideration! I tried to make Fluency as modular as possible so hopefully it shouldn't be too hard to add another engine. If you take a look at how DeepL works with Fluency, it exists as it's own class and Fluency doesn't know how or what happens in DeepL, and vice versa. Following that structure, you could create a Lingvanex class that Fluency sends data to. Matching DeepL method parameters when calling it and data structures it returns could make it pretty straightforward.

An idea I had would be to create a bootTranslationEngine() method that is called in ready() which would determine what engine/API key is available and instantiate the proper translator class in something like $this->translationEngine. That would replace $this->deepL in the Fluency module. I would make it a goal to abstract out any references to any brand of translator in the Fluency module's logic.

As long as getClientBootData() and translate() take and return the same structure of data you wouldn't even need to touch the front end.

I haven't really written any code that is collaborated on before so I hope that the commenting and overall structure makes sense. If you ever have any questions or want a shortcut to figuring something I did out then message me and we can chat.

 

 

[B3ta]

On 2/19/2021 at 7:22 AM, adrian said:

Thanks for your thoughts @FireWire - I think I'll be having to switch to Lingvanex sooner than later, so there might be a PR coming you way. Let me know if you already have any thoughts on how you would like to implement different translation engines so that I hopefully do it in a way that you're happy with.

HI @adrian, any news on Lingvanex PR? 

[adrian]

On 4/14/2021 at 10:02 AM, B3ta said:

HI @adrian, any news on Lingvanex PR? 

Sorry, because I don't actually use a translation service to translate content in the admin I took the easy way out and just implemented a direct curl request to the Lingvanex API.

[Mr. NiceGuy]

Very nice module. It actually also works without a paid account using the free api. The only thing one needs to do is to change one line as the api endpoint is different. 500000 characters for free per month which should probably be enough for many applications. 

classes/DeepL.class.php:    $reqUrl = "https://api-free.deepl.com/v2{$endpoint}?auth_key={$this->apiKey}";

 

I got a javascript error with the module though if I use a non-CK Editor text field (CK editor not found). The textarea field is in a repeater and I saw you fixed a bug related to this in the last release. Presumably the issues are related.  I am on the latest PW dev 3.0.179

 

Uncaught ReferenceError: CKEDITOR is not defined
    at HTMLAnchorElement.<anonymous> (fluency.js:257)

[FireWire]

On 6/5/2021 at 3:54 PM, Mr. NiceGuy said:

Very nice module. It actually also works without a paid account using the free api. The only thing one needs to do is to change one line as the api endpoint is different. 500000 characters for free per month which should probably be enough for many applications. 

classes/DeepL.class.php:    $reqUrl = "https://api-free.deepl.com/v2{$endpoint}?auth_key={$this->apiKey}";

 

I got a javascript error with the module though if I use a non-CK Editor text field (CK editor not found). The textarea field is in a repeater and I saw you fixed a bug related to this in the last release. Presumably the issues are related.  I am on the latest PW dev 3.0.179

 

Uncaught ReferenceError: CKEDITOR is not defined
    at HTMLAnchorElement.<anonymous> (fluency.js:257)

Sorry I've been away for so long. Deciding to buy a house, a puppy, and then remodeling said house while under deadline on deploying a new company API at work is an example of excellent decision making haha.

I didn't know there was a free API and that is awesome news. Might have been introduced since I last worked on the module. I'll be pushing these changes tonight for both the free API option as well as the JS fix. Thank you for your help @Mr. NiceGuy.

@B3ta adding engines is a longer term goal, there are still some refinements to be made before I can do that. Would definitely take a PR if anyone wants to tackle that.

[FireWire]

Fluency now supports the new DeepL free accounts in the latest release (v0.3.0) which also includes some bugfixes. Now live on the repo.

Thank you to @Mr. NiceGuy for notifying me about the new free accounts from DeepL. Very exciting to know that anyone can use Fluency for any project or to play around with at no cost.

I hadn't noticed but since Fluency was released in November 2020 DeepL's supported languages has climbed from 12 to over 25 which is really awesome. Looking forward to seeing what languages they add next. Fluency will always offer all languages that DeepL supports as it pulls this information from the API, so no module updates are ever required to take advantage of these.

Thanks everyone and bugfixes and feature requests are always appreciated over on the Github repo.

[B3ta]

@FireWire awesome work! 

Currently i'm having some shortcodes [[city]] and seems like i'm getting a strange result https://cln.sh/IztOtO 

Do you think it can be fixed to maintain the [[city]] tag?  Probably it can be solved by adding the glossary option as the desktop app https://cln.sh/tRCCzV

2. for longer text (over 5000 words it seems like the translations is getting stuck on the green screen https://cln.sh/1TObKx

[FireWire]

13 hours ago, B3ta said:

@FireWire awesome work! 

Currently i'm having some shortcodes [[city]] and seems like i'm getting a strange result https://cln.sh/IztOtO 

Do you think it can be fixed to maintain the [[city]] tag?  Probably it can be solved by adding the glossary option as the desktop app https://cln.sh/tRCCzV

2. for longer text (over 5000 words it seems like the translations is getting stuck on the green screen https://cln.sh/1TObKx

That shortcode issue you're seeing was totally my fault. I will push a fix as soon as I can complete it. It will maintain the tags and all other excluded strings properly after that.

5k words! I haven't translated anything that long. As a little test I created a text file with ~5k words in it and the file size was 28kb. The API has a request limit of 30kb and this is almost certainly hitting that ceiling. I need to build out a way for Fluency to analyze the size of the translation, split it up into multiple requests to DeepL, and the re-assemble the multiple translations back into one. Unfortunately this is a hard limitation by the DeepL API so the workaround will have to come from me. I'll put that on the to-do list.

[FireWire]

@B3ta New version pushed with fixes for the [[tag]] exclusion. All excluded strings set up in the module config should work now.

https://github.com/SkyLundy/Fluency-Translation/releases/tag/v0.3.1

Still need to work on the large content translation though.

[B3ta]

53 minutes ago, FireWire said:

@B3ta New version pushed with fixes for the [[tag]] exclusion. All excluded strings set up in the module config should work now.

https://github.com/SkyLundy/Fluency-Translation/releases/tag/v0.3.1

Still need to work on the large content translation though.

Thank you for your quick reply and fixes! 
I just sent you a paypal donation. 

[FireWire]

40 minutes ago, B3ta said:

Thank you for your quick reply and fixes! 
I just sent you a paypal donation. 

Many thanks! Let me know if anything else pops up.

[B3ta]

On 6/23/2021 at 4:42 AM, FireWire said:

Many thanks! Let me know if anything else pops up.

It seems like the the issue is popping up even with smaller text form, around 800 words even on the pro version of the API. 

It would be nice if you can take a look and see if it can get fixed.

 

Thanks!

[FireWire]

On 6/24/2021 at 9:00 AM, B3ta said:

It seems like the the issue is popping up even with smaller text form, around 800 words even on the pro version of the API. 

It would be nice if you can take a look and see if it can get fixed.

 

Thanks!

I've seen translations take varying times depending on length and language. Are you letting it work for a while?  I will test more with longer content and look into how DeepL handles it.

[B3ta]

@FireWire Do you think you can bring an update to module to translate from secondary language to main language?

Eg:  Site Main lang is DE and sec lang is EN , but the article writer writes the text in english and the client would be able to translate from EN to DE (main language) 

 

[FireWire]

6 hours ago, B3ta said:

@FireWire Do you think you can bring an update to module to translate from secondary language to main language?

Eg:  Site Main lang is DE and sec lang is EN , but the article writer writes the text in english and the client would be able to translate from EN to DE (main language) 

 

I thought about building that out from the start but wanted to focus on getting the core of the module working first in a way that matched the way ProcessWire's multi-language setup works where there is a primary language and secondary languages.

Including the general translation tool in the top menubar where any text can be translated from/to any language was a short term solution for that limitation. Once Fluency is able to mature and all of the bugs fleshed out then revisiting that would be an option.

[FireWire]

@B3ta Update on the errors with lengthy content. Unfortunately it's not an issue with the module from what I can see. I was able to find out where the limitation on length was down to the sentence.

When the API is used to translate it returns standardized responses that returns the translated content when it is successful, or errors that give instructions on what went wrong. Unfortunately the issue we are experiencing causes the DeepL API to return nothing (null) which means that their system experiences and issue but does not respond the way that they say it should.

I'm going to be sending in a support request to DeepL detailing the issue and see what they say. Will report back with updates but as of right now it looks like Fluency can only handle medium amounts of text.

[FireWire]

22 hours ago, FireWire said:

@B3ta Update on the errors with lengthy content. Unfortunately it's not an issue with the module from what I can see. I was able to find out where the limitation on length was down to the sentence.

When the API is used to translate it returns standardized responses that returns the translated content when it is successful, or errors that give instructions on what went wrong. Unfortunately the issue we are experiencing causes the DeepL API to return nothing (null) which means that their system experiences and issue but does not respond the way that they say it should.

I'm going to be sending in a support request to DeepL detailing the issue and see what they say. Will report back with updates but as of right now it looks like Fluency can only handle medium amounts of text.

@B3ta

That was all totally wrong. I was improperly making the DeepL API call with the wrong method which was totally my fault.

After working on the issue I've tested translations with a lot of content and have had success up to ~22,000 words (rich text in a CKEditor field). If your PW fields has more than that, I'm impressed haha. That will have to be the new ceiling and I'll engineer a solution if people start hitting that. When working with content up to those levels though I think DeepL may be being generous with their stated API request size limit so keep in mind that number could fluctuate- just hypothesizing though. You should be more than okay when translating 5000 words though.

Will push a new version soon and let you know when it is ready.

[FireWire]

New version released. Fixed issue when translating larger volumes of text. In testing I was successful in translating over 20,000+ words in one field. When you get to large volumes of content like that there is a very noticeable amount of time it takes to complete that request, but it does it.

Download the newest alpha here: Fluency 0.3.2

This should solve your issues @B3ta, let me know if you experience any issues.