module Fluency - The complete translation enhancement suite for ProcessWire

[ryangorley]

20 hours ago, FireWire said:

@ryangorley fix pushed to the development branch. Let me know if that doesn't solve it!

Thanks! I'll be able to test that tomorrow afternoon.

[ryangorley]

@FireWire Got delayed on my end, but I've uploaded and tested the update. It's working great. Thank you!

[ryangorley]

So, I've bumped into something worth considering.

When I added more languages to my client's site the existing pages inherited the untranslated URL slugs from the default language. So I've got to go to the Settings for each page and manually click the translate from English button for each language to pull a translated slug. That's burdensome, but it seemed to be working... until I had to start translating a bunch of blog tags, then some slugs wouldn't translate. I couldn't figure it out until I realized that Fluency must be pulling a DeepL translation from the English slug, not the English title. You'd think it wouldn't matter, but it does. Here's an example:

English > French:
AI App > Application IA
ai-app > ai-app

It turns out DeepL translates ai-app to ai-app in every language. I don't think it knows what ai-app means so it just returns it.

I don't know how commonly people will run into this problem, but I've been running into it a bit. Sometimes DeepL will return the slug unchanged, sometimes it will provide a slightly different translation than it did for the title. My suggestion would be for the slug translation prompt to use the default language page title when calling for a translation. I could see where pulling the title could create other surprises, such as when someone has the page title "Contact Us" but the slug "contact" they'll be getting longer slugs returned. Still I think that would be preferable to DeepL just silently failing. Maybe others feel differently. It's worth consideration though. 

[ryangorley]

I'll add, perhaps for @ryan to consider, that an equally good solution would be for ProcessWire to borrow a behavior from elsewhere. In WordPress if I want to reset a slug I just clear the field and save the post. It then auto-populates that field from the post title as it does when the post is first created. In ProcessWire clearing the slug fields and saving the page just reverts the fields to their prior state. A small, but nice feature in WordPress I'd love to see in PW. :)

[adrian]

1 minute ago, ryangorley said:

I'll add, perhaps for @ryan to consider, that an equally good solution would be for ProcessWire to borrow a behavior from elsewhere. In WordPress if I want to reset a slug I just clear the field and save the post. It then auto-populates that field from the post title as it does when the post is first creation. In ProcessWire clearing the slug field and saving the page just reverts the fields to their prior content. A small, but nice feature in WordPress. :)

You might find this useful: https://processwire.com/modules/page-rename-options/

[update AG]

Does anyone has a problem with installing Fluency (1.0.7) in Processwire 3.0.229 under PHP 8.2?

I can install it but when it comes to selecting the preffered engine I get a 503 error.

I suspect a mod_security issue since the module wants to save a serialized string (see below) to the database, when selecting the engine.

Anyone run accross this problem and knows a workaround? Maybe @FireWire can help?

Edited February 8 by update AG
PHP is 8.2 not 8.1

[FireWire]

On 2/7/2024 at 7:00 AM, ryangorley said:

I don't know how commonly people will run into this problem, but I've been running into it a bit. Sometimes DeepL will return the slug unchanged, sometimes it will provide a slightly different translation than it did for the title. My suggestion would be for the slug translation prompt to use the default language page title when calling for a translation. I could see where pulling the title could create other surprises, such as when someone has the page title "Contact Us" but the slug "contact" they'll be getting longer slugs returned. Still I think that would be preferable to DeepL just silently failing. Maybe others feel differently. It's worth consideration though. 

This is interesting and I hadn't thought of that before. I'm no expert on DeepL's internal logic, but I'd venture to guess that this "passthrough" is also what prevents it from trying to translate things like proper nouns or selections that it can't translate with confidence in accuracy. Double edged sword I guess.

Edge case- the page title could be "Take The Next Step" but the slug in English is "/contact-us/" so the translation to German would be "/de/den-nachsten-schritt-tun/" which doesn't match. It becomes more complicated when someone renames the page to "Find Out More", so the German slug is orphaned entirely. I'm not sure how prevalent that would be, but it draws a connection between page title and URL where they serve different roles.

I haven't seen this situation myself so I don't have anything to go off of or know if this is a big ask- is it possible to tweak the default language URL so that it has more parity across languages?

2 hours ago, update AG said:

I suspect a mod_security issue since the module wants to save a serialized string (see below) to the database, when selecting the engine.

This is a very interesting situation and one that I haven't heard come up or experienced myself. I am at a bit of a disadvantage because I haven't used Fluency in production since I released the rewritten module last year and that DB serialization was only introduced then. Do you have access to any logs where you could verify a mod_security issue? Is there anything in the response headers of the 503 page that could help? I can modify the the module but would like to be sure that it's not a different problem that may need to be addressed.

[update AG]

1 hour ago, FireWire said:

This is a very interesting situation and one that I haven't heard come up or experienced myself. I am at a bit of a disadvantage because I haven't used Fluency in production since I released the rewritten module last year and that DB serialization was only introduced then. Do you have access to any logs where you could verify a mod_security issue? Is there anything in the response headers of the 503 page that could help? I can modify the the module but would like to be sure that it's not a different problem that may need to be addressed.

I managed to get the Fluency Module installed and the settings set correctly. I installed in on another server a copied the settings directly from the database. So, that is that. But the Module is still not working. It looks like the translation-api page cannot be accessed. I get an "Unrecognized path" error (see screenshot), when I access the page /processwire/fluency/api/translation/ directly. Since the whole thing is working on another server it must have to do with some server restrictions but I can't figure out what it could be. Any ideas?

[FireWire]

7 minutes ago, update AG said:

when I access the page /processwire/fluency/api/translation/ directly

This path is only for use by AJAX to make the translation request, so accessing that URL directly won't work.

8 minutes ago, update AG said:

Any ideas?

Can you open the developer tools and go to the Network tab, try to translate something and then show me what the response body is?

[update AG]

10 hours ago, FireWire said:

This path is only for use by AJAX to make the translation request, so accessing that URL directly won't work.

Can you open the developer tools and go to the Network tab, try to translate something and then show me what the response body is?

You can access it directly and get a JSON response, when the path is correct: 

{"error":"FLUENCY_METHOD_NOT_ALLOWED","message":"Fluency API Error: Method Not Allowed"}

But I didn't get anything so I looked closer to the path and saw that I had a wrong path. I had /processwire/fluency-1/api/translation/ as api-path, which was probably due that I deleted the module and reinstalled it so maybe it created this new path. I changed it now and it works.

Second: Still when I try to edit the settings page in the Module I get the 503. See the attached network response. I still think it is somehow a security issue, when trying to save to the database due to all this serialized data in the POST-request.

DB is by the way MySQL 8.0.

[FireWire]

8 hours ago, update AG said:

I still think it is somehow a security issue, when trying to save to the database due to all this serialized data in the POST-request.

Thank you for double checking. I've seen a lot of different mod_security implementations (have done it myself) and there can be a lot of variance. I had one host change the error response to HTTP 418 I'm A Teapot (very unhelpful...) and others return other 4xx codes, but haven't seen a 503 Service Unavailable because that indicates a server failure, not an issue with the request. That's not to say that your hosting provider hasn't chosen a 5xx status and has a really strict implementation. Alas, I'll work on updating how the module stores information and push to a separate branch so you can test and see how it works in your environment.

I'm putting out a few fires today (anyone else having to deal with DMARC records?) but I'll get to that as soon as I can spare the time.

[FireWire]

Hey @update AG I've pushed a revision to the module that changes how configuration information is stored in the database and removed all serialization. It's on a separate branch so that you can test it. This requires that any Translation Engine related data will need to be re-configured.

Please download and test from this branch on the Fluency repository.

[update AG]

On 2/11/2024 at 2:31 AM, FireWire said:

Hey @update AG I've pushed a revision to the module that changes how configuration information is stored in the database and removed all serialization. It's on a separate branch so that you can test it. This requires that any Translation Engine related data will need to be re-configured.

Please download and test from this branch on the Fluency repository.

Hey @FireWire Whatever you did, it's working now: Downloaded your branch, deinstalled the module first and than installed your new version. All is working fine. Configuration of the settings now all work without getting an error anymore. Great help. Thanks a lot.

[FireWire]

@update AG Great to hear! I changed how Fluency stored config data. The issue you experienced helped find an edge case where the module may cause issues for some people. Your issue where you experienced two translator pages has also been fixed as well. Appreciate the feedback and helping make the module better.

I'm going to add this to the next release but will try to see if I can prevent any need to uninstall/reinstall and preserve settings when upgrading.

[FireWire]

Hey all!

Fluency 1.0.8 has been released. Now available for upgrade in ProcessWire admin and in the modules directory. For the Composer fans out there, Fluency has been added to Packagist and can now be installed by running `composer require firewire/fluency`

This version contains a new feature where you can now disable translation on a per-field basis. Just check the "Disable translation for this field" checkbox under the 'Details' tab and Fluency will not be added to that field.

This is helpful where fields may contain different content for different languages but may not be suitable or desired for translation- email addresses, phone numbers, full string URLs, etc. It can help clean up the UI when editing pages and prevent unnecessary translations.

Also includes the following:

• Activity overlay text has been resized and reformatted, now looks better where the size of the field is smaller.
• Translation cache is now enabled by default as stated in the README file
• Performs some polite cleanup when uninstalling the module.
• Fix issue where CKEditor fields in collapsed Repeater and RepeaterMatrix fields may fail to initialize. Credit to @ryangorley for finding/reporting.
• Fix issue where fields with combinations of languages that are and are not configured in Fluency are properly handled. Languages that aren't configured in Fluency are now indicated where others have the translation button present. Credit to @ryangorley for finding/reporting.
• Fix for issue where the way that Fluency stored module configuration data may cause ModSecurity false positives depending on how rules were configured on that server. Credit to @update AG for finding/reporting.
• Various minor fixes and code cleanup.

Please share any bugs here or, if possible, create an issue in the Fluency Github repo. Thanks!

 

[theoretic]

Hi friends! Many thanks for this brilliant module!

It appears I've found a heavy issue concerning php 8.3.7 . In brief: Fluency crashes, giving multiple errors of this kind:

Uncaught Error: Undefined constant "Fluency\Engines\DeepL\CURLOPT_CONNECTTIMEOUT" in site/modules/Fluency/app/Engines/DeepL/DeepLEngine.php:220

Digging inside reveals the following code sample:

    $requestConfig = [
      CURLOPT_CONNECTTIMEOUT => 5,
...

CURLOPT_CONNECTTIMEOUT looks here like a constant, however it's not. PHP 8.3.4 can however handle this kind of arrays, and Fluency works like a charm. But it's not the case for PHP 8.3.7 .

I think it's worth taking this into consideration during further development. I've also created an issue on the Github module page.

Hope this will help to make this module even better 😉

[FireWire]

@theoreticHey there- we figured out the issue on Github but wanted to mention here for others as well.

Fluency uses CURL to make API requests to the translation service so that is required to be present on the server. I'll note this in the documentation on the next release as a requirement for the module since it's possible to use ProcessWire and never need or use CURL.

Good catch and thanks for the report!

[bernhard]

@FireWire have you considered using WireHttp for the requests?

[FireWire]

@bernhard I did try to use it at one point. I remember that it wasn't compatible with how DeepL required it's URL parameters to be structured. I think it was the way that it had multiple GET variables with the same name and that didn't serialize properly when using WireHttp.

That may be a moot point now because the DeepL API now accepts JSON encoded POST requests so that URL issue isn't a problem.

Prior to that I ran into an issue with WireHttp not handling 404 responses in a way that I need to rely on, so it isn't generally top of mind. Long story short, if a 404 is returned, WireHttp will ignore that valid response and try again.

As an aside- I'm working with an API right now where 404 is a valid response for a record that doesn't exist, so trying 2x doubles the API credit usage with that service. May or may not be an issue in this instance, but it is something to keep in mind when HTTP requests have a hard cost associated with them.

[Martin1]

Hey @FireWire

First of all, thanks a lot for this amazing module!

I'm currently working on a project with Fluency + DeepL that needs the glossary feature: https://developers.deepl.com/docs/api-reference/glossaries/openapi-spec-for-glossary-management

Since glossaries created on the DeepL website cannot be accessed by the API, they need to be sent to the API with ProcessWire, and translation calls need to have a param with the glossary id for every language pairs.

I have most of the work done, and I'd be willing to share it once I clean it up, however I'm not sure how to go about this. My first instinct was to do a stand-alone module with Fluency as a dependency, however I can see that it is heavily locked down with final classes + private and unhookable methods.

I'm wondering if there's some interest for this, and if yes, you'd rather have a pull request with the DeepL Glossary feature baked-in, or a pull request which would open up some classes / methods to be extendable, which would allow me to make a new module?

Thanks!

[FireWire]

@Martin1 Great idea! Thinking out loud, I'm considering a couple of things by way of how to implement.

At it's core, Fluency is intended to provide like-kind behavior that is similar enough between current and future translation services/engines so that the module overall doesn't stray into having over-specificity for functionality. Like having general module methods that reach too far into features or behavior of specific engines. So my first thought is an add-on module to supplement DeepL's abilities- but this is open source so contributors should definitely have input that can make it better. Some thoughts below on addressing the issues you listed-

I think that Google Cloud translate may have a similar feature but it would come down to how well the behavior could be abstracted to provide a common enough interface at the engine level where a common Data Transfer Object, and thus a feature interface, can be shared.

Final classes- the DataTransfer Objects are definitely a "hard" final class where each should inherit from the base abstract DTO.  There is some extra flexibility built in with the ability to add a 'meta' property that can hold some engine-specific behavior, like "formality" which DeepL supports but others like Google don't. Other instances of removing final are probably on the table, I'm not sure I should have made the Fluency module class final. A glossary would definitely warrant it's own DTO in the same spirit as the EngineApiUsageData, EngineTranslationData, EngineLanguagesData, etc. represent data passed between domains for individual features. Otherwise it's probably just a matter of following the Principle of Least Privilege.

Unhookable methods- I don't good excuse for this one. Basically left out because hookable methods don't allow for named arguments. This is 100% open to change and was pretty short sighted on my part given that this is a module after all... I should just push a release that does adds hooks regardless.

Private methods- This pretty much just follows the Principle of Least Privilege again and, like the final classes notes above, many are open for changing that.

All said, I'm kind of open here as far as integrating it into Fluency or opening up class extension, public-ifying methods, and adding hooks. All said, is there one approach that feels better to you?

[Martin1]

If you want to avoid the extra burden of maintaining this on multiple translation services, I think it's best as an add-on. Although I've made everything baked-in for now, I'm looking at the api reference for google and we could abstract and make a DTO for both, however it looks like the implementation would be way more complex on google's side, starting by the fact that the endpoint is not even the same api version.

The more I think about it, the more I think I'm overcomplicating things. The way I see it that would be the easiest: I would make a new client for deepl and can already access the account type and api key from your package's config. I would not need to hook into Fluency for managing it. Anything done programmatically to translate, I already can use the public translate() method by adding the glossary id to the options['requestParameters']

The only thing I need to hook on in Fluency would be the "Translate to all languages" field link and rebind a new ajax call with the glossary id to the request parameters, I have not tried but it seems feasible.

So then there should not be any need to make anything public or hookable!

[FireWire]

That sounds pretty great.

3 hours ago, Martin1 said:

The only thing I need to hook on in Fluency would be the "Translate to all languages" field link and rebind a new ajax call

This is probably the most annoying part, all of that is rendered on the client side in JS. I wish I rendered it client side by hooking into the render methods for the inputfields, but I didn't think about that at the time. So the challenge would be waiting until the translation buttons are rendered first, which is pretty fast because it doesn't involve any requests to initialize. The only delay is for the js file itself to load and the elements to be appended in the DOM.

I really want to make some internal improvements to Fluency, especially as nifty things like this come about so the module can be more easily worked with and extended. If I can be of any help or if there's any questions you have while slogging through my code, I'm here to help haha. PRs are open if you end up finding some things that can be improved anyway. Looking forward to seeing what you come up with!