Search the Community

Perhaps the module can better communicate this in some way. I'll think about ways to possibly remind superadmins if translation is not fully ready. Google has more languages, DeepL has a few extra features. The extra features in DeepL have defaults that you only need to change if you really want to, so no additional configuration. I prefer the translations from DeepL, but Google is a good alternative to have because of their language selection. I've had some challenges with this myself. I usually create a dev key that is unrestricted for development since I know that it won't be used anywhere else and a separate API key for production where the IP address will be stable and the setup is more permanent. No problem at all.

It was within the class that I tested it, and I did get an error when using that snippet in a template. However, happy to confirm the mistake was my own. I started a completely fresh site to test it, and was using Google Cloud. Clearly the first time I didn't notice the popup in the module settings telling me that authentication had failed 🤦‍♂️ I assumed Google just had less functionality than DeepL and therefore less configuration. I'm not sure why I regularly run into this issue, but when restricting the API key in Cloud Console to website URLs, it seems a bit unreliable. Or maybe it just doesn't work with local addresses. Of course, nothing to do with your module, which I'm happy to say is working now and looks great. Sorry to have troubled you with it!

That would be nice. To be honest, I never use forums or website search. By default, I just fire up Google. My usual suspects are /talk /blog /docs site:processwire.com/ deepl site:processwire.com/talk deepl site:processwire.com/blog languages site:processwire.com/docs $page

Maybe we can extend the site-wide search to include the body / description of modules as well, be it with less priority? This is an example of a use case: I knew a module exists that uses DeepL to translate site content, but when I searched for DeepL I got 0 results: https://processwire.com/search/?q=DeepL. However, DeepL gets mentioned 5 times in the body of this module page. It was only until I tried searching for terms like 'Translate' that I could find this module. Edit: there is another module that I missed and in my opinion should have popped up.

@Tiberium I have another way to attack this so I'll put together another commit on that branch and let you know. Unfortunately adding an ability to manually assign languages like the TranslatePage module would be a heavier lift than you might think since there are more features that Fluency uses. It would require rewriting the module config page for DeepL. The true solution is to make sure that the module performs correctly for everyone and preserve the UI that people are familiar with. It wouldn't be a good move to require some people who are having difficulty with API calls to forego features that are available to others who aren't. So I want to get this right for everyone 👍 Hold tight, I'll come back with another test. Please expect that via private message so we can more easily converse without adding too many more posts to this thread. When the issue is solved I'll come back and post a new release.

Hmm I was about to test it, on the specific customer project, who had that problem and now I'm hit on the Problem in the plugin configuration screen. So Fluency will not more let me connect the languages from ProcessWire to the DeepL one, and so I can not test it. Sorry. If it helps, the TranslatePage (via DeepL) - Plugin on the other hand, has no problems with the same API-Key.

@snck The purpose of the requests on the config page are to confirm that your API key works by making an API request to get the available languages. The only time that it will make 4 API calls on the config page is when you are first entering your API credentials. After you have entered valid credentials it will only make 2 API calls when you load the module config page. This hasn't ever been an issue before so I'm very curious why you are seeing a 429 but others haven't. Here's some additional details on why you are seeing 2 or 4 API calls for languages in Fluency when using DeepL: Fluency needs to know what languages a service allows you to translate from and what languages you can translate to. Google Cloud Translate will give you this information in 1 API call. DeepL requires you to make two API calls, one for source languages and one for target languages. Fluency is built with a modular architecture so that new translation services can be added easily. If you're interested you can compare DeepLEngine.php with GoogleCloudTranslationEngine.php. Each have the same methods that return the same data type but how it gets that data is entirely internal. To see what I mean about the 2 API calls vs 1 API call compare the getLanguages() method in each of those. 2 API calls is technically just one execution of the getLanguages() function. You should only see 4 API calls when: You first enter your API credentials and hit save You change translation engines You entered incorrect credentials and have to re-enter them After that it should only be 2 when using DeepL in Fluency. Back to your issue- 4 API calls shouldn't cause a 429. The DeepL team states that API calls should be limited to 50 requests per second. That comment is in an issue in their Node library, but the limits should still apply for all API usage. I just removed my credentials and re-added them without any issues. Are you seeing 4 API calls every time you load or save the module config page? I have to ask the obligatory security question- is there any chance that your credentials are being used by someone else or were compromised? I know that you said that you just created your API key today but you would have to be absolutely hammering the DeepL API to get a 429. I've never seen it during regular usage and it hasn't come up here in the forums by anyone who has been using the module for years. This is a tough issue to crack especially since it's happening for you but not me at the same time. Let me know if there's any additional info you can share.

@FireWire No, never used Fluency or the DeepL API before. The account has been created today, the key is fresh and I could successfully use it with curl from the shell without problems. I do not really understand the goal of the requests that are made on the settings page. As I get status 200 for 3 times, the connection to the API looks fine to me. This looks like requests 2 and 4 are identical (same endpoint, data and method). As they are made immediately after another a rate limit error is to be expected. What is the outcome or behaviour you would expect?

Yeah, that's DeepL returning 429 as your error and nothing as the response body. I'm at a loss on how to diagnose that. Have you been able to translate content successfully in the past? Have you used up your allowable number of translated characters? This is going to have to be something on the DeepL side and I don't know if there's anything I can do to help out. Like I mentioned, I just translated using my API key and things are working alright.

@snck That may be a gamble... I think I've seen some situations where DeepL responses aren't very descriptive beyond the HTTP code that they return which can be frustrating. Unfortunately when I test my API key (free) I'm getting translations successfully. In your case the only way to get "Translation rate limit exceeded" is if DeepL returns an HTTP 429 code so that is another message that Fluency is relaying to you from DeepL and not originating with anything in the module. Whether the HTTP 429 applies in your case, I don't know. You may want to check the API key type that you have and ensure that it is valid for making translation requests. If you would like to see what DeepL is responding with directly and you aren't afraid to dig into some code then you can dump the response at L252 in DeepLEngine.php. Because the translations are made via AJAX in ProcessWire, you'll have to use the developer tools in your browser to view the response after clicking the translate button for a field. If you're using Tracy then adding bd() is the best way to see the result in your browser. Sorry I can't be of more help, there have been some instances in this thread where people have had some hiccups with DeepL. It doesn't happen very often and when you find the cause/solution for your issue here it should be smooth sailing.

@FireWire Thanks! I had already checked that. I picked "Free" (which is correct) and inserted a valid API key that has never been used before. I only get the "Translation rate limit exceeded, please wait then try again" error (see below). The fluency-engine logs show the following message: Engine: DeepLEngine /Error: RATE_LIMIT_EXCEEDED /Message: No Message /Response: null Is there any other place where I could look for more verbose information? I tested the API key and made sure it is correct: curl -X POST 'https://api-free.deepl.com/v2/translate' -H 'Authorization: DeepL-Auth-Key [MY API KEY]' -d 'text=Hello%2C%20world!' -d 'target_lang=DE' {"translations":[{"detected_source_language":"EN","text":"Hallo, Welt!"}]}%

@Mike-it some steps: Double check that you've picked the correct "Free" or "Pro" subscription type on the module config page. This is an issue with a DeepL account, API key, or a Free/Pro setting. If you receive that error again, next to the "check credentials" there is a +2, click on the chevron-down and see what the additional message is that may be hidden. There is only one place in the module that can return that error message and it is based on the HTTP response status. So in that case, DeepL is indeed returning an HTTP 403 and Fluency is just relaying the information. The error message you're seeing is the "polite" message from Fluency. Any errors returned by a translation service are logged so there may be a more specific message in the fluency-engine log if DeepL provided one.

Hello everyone, I'm excited to announce that Version 1.0 of ProcessTranslatePage is now available on GitHub: https://github.com/robertweiss/ProcessTranslatePage Please note that there are some breaking changes in this update. I recommend uninstalling and then reinstalling the module to ensure everything works smoothly. You can use the same DeepL API key that you use with Fluency. After installation, you will (hopefully) see two new fields for the language template. However, only the ⁠'translate_locale' field is required for the module to function properly. Thanks for your patience, I hope you find this update useful!

Hi @Ivan Gretsky and @FireWire, Sorry it took me so much time to respond to the big Fluency update. To be honest, I already refactored my module a couple of months ago to work without any dependencies (except the DeepL API), so Fluency is no longer necessary to use it. Since I only did this refactor for one client, I need to remove some customizations before I can push the new version to Github. I don’t expect this to take too long, so I’m hopeful that I’ll be able to release Version 1.0 for you to test tomorrow or thursday 😊. Thanks for your patience!

@bernhard Apologies for the delay! I've pushed updates to the dev branch. I had a few changes that were already going into the next version that I had to do some extra testing for and a lot of housekeeping. Let me know if this works as well for you as it did me and I'll push another version to main https://github.com/SkyLundy/Fluency/tree/development New feature that some may be interested in- adding DeepL languages for translation that are technically supported by their API but not delivered by the languages endpoint yet. You can read more about when that happens and why on this DeepL documentation page. To manually define languages in Fluency that are supported for translation, you can use a hook. This will make the language available both within the global translation tool as well as available for configuration with a ProcessWire language on the module config page. <?php namespace ProcessWire; // ready.php use Fluency\DataTransferObjects\{EngineLanguageData, EngineTranslatableLanguagesData}; // Hook after Fluency gets the available languages from the DeepL API wire()->addHookAfter('Fluency::getTranslatableLanguages', function (HookEvent $e) { // $e->return is an instance of EngineTranslatableLanguageData // The `languages` property returns an array of EngineLanguageData objects $languages = $e->return->languages; // Create a new data object, define the values according to the API docs, add it to the original languages array $languages[] = EngineLanguageData::fromArray([ 'sourceName' => __('Arabic'), 'sourceCode' => 'AR', 'targetName' => __('Arabic'), 'targetCode' => 'AR', 'meta' => [ 'supports_formality' => false, // This is determined by DeepL, check to see if it's available in their API docs ], ]); // Instantiate a new EngineTranslatableLanguageData object with the language array then assign it to the return value $e->return = EngineTranslatableLanguagesData::fromArray([ 'languages' => $languages ]); }); @Hari KT This is a new method supported by Fluency to add Arabic to Fluency before DeepL lists it for translation. Thank you for inspiring a solution with your Github issue on languages not being available in Fluency. This is available on the dev branch of the Fluency repo if you would like to help test it out @ryangorley This will address the issue you had with Traditional Chinese, albeit a lot later than may be useful for you, but the hook above can be modified to resolve that problem. I'm surprised how long the language had been listed as "available soon" on their API documentation 🤷‍♂️ I would have gone this route sooner had I known there was some documentation about this, I don't think it existed at the time.

Hi @FireWire First of all, thank you for the module and all the work you have put into it! I just wanted to test the module in a real world project, but quickly realised that certain terms are always translated incorrectly by Deepl. In the web version this is easy to solve with the personal glossary. Is there also a way to create a glossary here, or to tell the API in some way to use my glossary from my account?

@MarkE I know this is a late answer, but I think this is a great question and something that can feel daunting. Hopefully it's still helpful for you and possibly others. I originally wrote my translation module back in 2020 and when it came time to build upon it, add new features, and respond to feedback from the community that first version felt like a straightjacket in terms of structure. I rewrote it last year and have been able to iterate on it much more easily and confidently. This is really a decision based on how you view your project and what "feels good". When you get to writing modules as complex as the one you're describing, as always, the best way to reason about your approach is consistency and maintainability. This is more critical than trying to fit your code into someone else's structure. The most important way to think about organization is considering where you would first look to find a specific feature or behavior and if there are enough of these to warrant their own directory. My translation module probably isn't as complex as yours is, but it may not matter because we're both writing software applications that must be understood and maintained. Anyone can view the repo I shared above, but here are some notes about my reasoning. The purpose of this isn't meant to be a set of instructions or a roadmap, but more of a way to help you think about what is best for your application. [app] - This contains everything that supports what the module must do. If it has logic or supports logic, it goes here. [Caching] - Caching is a performance requirement and required abstraction and logic to tailor PW's caching feature for my use EngineLanguagesCache.php TranslationCache.php [Components] - Fieldsets used in one or more places where abstracting helped keep clean code in other files, like the main Module Traits - Shared behaviors used by Component files FluencyApiUsageTableFieldset.php FluencyStandaloneTranslatorFieldset.php [DataTransferObjects] - All DTOs that encapsulate data provided to and returned by methods in different classes Traits - Shared behaviors used by DTOs AllConfiguredLanguagesData.php ConfiguredLanguageData.php DEVDOC.md - A guide for me and others about how to use DTOs and their purpose as general data objects EngineApiUsageData.php .... [Engines] - Self contained "submodules" that interact with 3rd party services, both current and future. [DeepL] [GoogleCloudTranslation] [Traits] DEVDOC.md FluencyEngine.php FluencyEngineConfig.php FluencyEngineInfo.php [Functions] - General functions imported to support specific sub-behaviors that serve a specific purpose in more than one-two places fluencyEngineConfigNames.php typeChecking.php [Services] - Features that are added to Fluency that extend beyond the primary function of translating page content FluencyProcessWireFileTranslator.php FluencyErrors.php FluencyLocalization.php FluencyMarkup.php [assets] - Files that contain admin-facing UI files and compiled code, Gulp compiles and outputs to this directory [img] [scripts] [styles] [src] - Source JS and SASS that gets compiled and output to the assets directory [scripts] [scss] ...(editor/dotfiles) CHANGELOG.md - Semantic versioning is your friend, this is as important for you as it is for others Fluency.info.php Fluency.module.php FluencyConfig.php LICENSE README.md - This goes into high detail of all features because it also organizes my understanding of everything Fluency does ...(JS module configs and Composer file) I spent a lot of time on this decision. Probably overthought it, and since then there are things that I would do differently- however, how it exists now has been great to work with. My approach was entirely based on what this module does now, and what it may do in the future. The focal point and goals of this approach serve these purposes: - Fluency.info.php - Not required to separate, but helped keep the main module file cleaner and purpose-driven - Fluency.module.php - Aims to primarily contain methods needed to integrate with ProcessWire and any methods available via the global $fluency object - FluencyConfig.php - ProcessWire needs this to be configurable, but the amount of logic it contains certainly benefits from its own file Things I like having lived with it for over a year now. The directory structure is great and supports future expansion since it organizes by purpose. The specific way I structured my subfolders may not make sense for most others, but the message here is the logic behind the approach. Like I said, it seemed overengineered at the time, but now I know exactly where everything is and where everything should be in the future. This module has heavy UI integration and the way the JS files in src/ are broken down makes it clear what everything does. Because JS can import modules from other files, it really doesn't matter how many you have. The JS files for each Inputfield has on more than one occasion enabled me to add features or fixes that would have taken much longer without the granularity I put in place. Initially this module only supported DeepL for translation, but because of how Engines is structured as individual units, it was relatively easy to come back and add Google Cloud Translation because of how the individual units of behavior were organized and supported. What would I do differently having lived with it for over a year now? I would change the file names in app/ and remove the 'Fluency' prefix. That's why we have namespaces. Name collisions are handled by PHP so a simple file naming approach would have been best. This is something that I could change though without impacting the end user. FluencyErrors, FluencyMarkup, and FluencyLocalization might be better located in a 'Support' directory or something, but that would just be applying my approach to organization rather than needing to serve a hard purpose. It doesn't cause any confusion or problems as is. Probably some little things here or there, but not nagging enough to mention or even remember. Anyway. I chose a directory heavy structure with nesting which was a personal choice. There are a lot of great modules out there with different structures. A mark of good organization is asking the question "if I've never looked at this codebase before, would it be something I could get up to speed quickly and contribute without difficulty?". This may seem like a disconnected and impersonal approach, but inevitably you will feel like you need to relearn your own codebase in the future if you've spent enough time away from it. This is a great place to start. In a complex enough project, it may be worth considering separate modules that are installed as dependencies. This would let you group like-kind behavior where it may also be possible that some separate functionality could stand on its own. If you can organize things to separate concerns enough, it's trivial to access other installed modules from within modules using $this->wire('modules')->get('YourOtherModule'); and access all of its features and functionality. Since you can configure your module to have required dependencies, you can be sure that the modules you need will always be available. Organizing by purpose and features is really helpful. If the behavior doesn't provide anything useful as a standalone, then consider keeping it within the "wrapper" module. It depends on how portable you need individual behaviors to be separate from others. I'm not sure too much on the specifics of the implementation, but these are files that perform a specific type of work and would be candidates for their own directory. There's no rule for this, but it makes sense because it gives you a single place to look for these files. Long story short, trust yourself now and think about future you who will work on this later. You'll thank yourself for it. Hope this was relevant enough to be helpful even though it's a late-in-game response.

@FireWire I feel bad, because there is a note at the top of the DeepL documentation page that says Traditional isn't yet available via the API, but I guess it will be at some point. Sorry for the false alarm, and thanks for looking into this!

@ryangorley I'll leave that response up in case it helps anyone else but I just took a look at the DeepL website source code and it shows that it translates into Traditional Chinese which is ISO code zh-Hant, but the API is returning zh-Hans which Simplified Traditional Chinese. I think this may be where the disparity may be coming from. Here's the HTML used on their home page translation tool. They are providing full Traditional Chinese translation via the web, but not via the API (no idea why) <div data-testid="translator-target-lang" dl-selected-lang="zh-Hant" class="mobile:hidden truncate">Chinese (traditional)</div> So your best bet is to use zh-Hans in Fluency as that's the closest they offer via the API. I jumped onto an installation of Fluency on my own to check that out but jumped the gun with the translation cache reply...

Hey @FireWire, thanks again for this great module. Quick question. I'm not seeing Traditional Chinese as an option. It was recently added to DeepL. Does that need to be enabled in some fashion on your end?

You can use Google Translator Bar, but the result is usually much worse than that of the suggested Fluency with Deepl https://www.w3schools.com/howto/howto_google_translate.asp

Hey @FireWire Probably a great module...? I finally had the time to explore your module but I keep running into the same Deepl API errors and I don't know where to look for a solution or what to do next. This is PW 3.0.229 on PHP 8.2 Fluency 1.0.9 on a pretty basic PW testsite. Deepl API key shows as valid and active and in my Deepl account it says: Access to DeepL REST API When I click on translate the following happens: The error log shows: Engine: DeepLEngine /Error: BAD_REQUEST /Message: Invalid request: Expected 'text' parameter to be an array of strings /Response: {"message":"Invalid request: Expected 'text' parameter to be an array of strings"} The TRANSLATION modal shows only the word undefined in the target language. It's just a testsite, so no rush. I would be greatful for any advice.

If you want to avoid the extra burden of maintaining this on multiple translation services, I think it's best as an add-on. Although I've made everything baked-in for now, I'm looking at the api reference for google and we could abstract and make a DTO for both, however it looks like the implementation would be way more complex on google's side, starting by the fact that the endpoint is not even the same api version. The more I think about it, the more I think I'm overcomplicating things. The way I see it that would be the easiest: I would make a new client for deepl and can already access the account type and api key from your package's config. I would not need to hook into Fluency for managing it. Anything done programmatically to translate, I already can use the public translate() method by adding the glossary id to the options['requestParameters'] The only thing I need to hook on in Fluency would be the "Translate to all languages" field link and rebind a new ajax call with the glossary id to the request parameters, I have not tried but it seems feasible. So then there should not be any need to make anything public or hookable!

@Martin1 Great idea! Thinking out loud, I'm considering a couple of things by way of how to implement. At it's core, Fluency is intended to provide like-kind behavior that is similar enough between current and future translation services/engines so that the module overall doesn't stray into having over-specificity for functionality. Like having general module methods that reach too far into features or behavior of specific engines. So my first thought is an add-on module to supplement DeepL's abilities- but this is open source so contributors should definitely have input that can make it better. Some thoughts below on addressing the issues you listed- I think that Google Cloud translate may have a similar feature but it would come down to how well the behavior could be abstracted to provide a common enough interface at the engine level where a common Data Transfer Object, and thus a feature interface, can be shared. Final classes- the DataTransfer Objects are definitely a "hard" final class where each should inherit from the base abstract DTO. There is some extra flexibility built in with the ability to add a 'meta' property that can hold some engine-specific behavior, like "formality" which DeepL supports but others like Google don't. Other instances of removing final are probably on the table, I'm not sure I should have made the Fluency module class final. A glossary would definitely warrant it's own DTO in the same spirit as the EngineApiUsageData, EngineTranslationData, EngineLanguagesData, etc. represent data passed between domains for individual features. Otherwise it's probably just a matter of following the Principle of Least Privilege. Unhookable methods- I don't good excuse for this one. Basically left out because hookable methods don't allow for named arguments. This is 100% open to change and was pretty short sighted on my part given that this is a module after all... I should just push a release that does adds hooks regardless. Private methods- This pretty much just follows the Principle of Least Privilege again and, like the final classes notes above, many are open for changing that. All said, I'm kind of open here as far as integrating it into Fluency or opening up class extension, public-ifying methods, and adding hooks. All said, is there one approach that feels better to you?

Hey @FireWire First of all, thanks a lot for this amazing module! I'm currently working on a project with Fluency + DeepL that needs the glossary feature: https://developers.deepl.com/docs/api-reference/glossaries/openapi-spec-for-glossary-management Since glossaries created on the DeepL website cannot be accessed by the API, they need to be sent to the API with ProcessWire, and translation calls need to have a param with the glossary id for every language pairs. I have most of the work done, and I'd be willing to share it once I clean it up, however I'm not sure how to go about this. My first instinct was to do a stand-alone module with Fluency as a dependency, however I can see that it is heavily locked down with final classes + private and unhookable methods. I'm wondering if there's some interest for this, and if yes, you'd rather have a pull request with the DeepL Glossary feature baked-in, or a pull request which would open up some classes / methods to be extendable, which would allow me to make a new module? Thanks!