Emails in IDN format do not get saved, cannot be entered

[gebeer]

Hi all,

I'm on PW 3.0.209 and cannot save international Email strings (like "hans.müller@müller.com") to email fields through the GUI and also not through the API. Since 3.0.208 $sanitizer->email supports IDN with option "allowIDN". But the accepted values cannot be saved.

As far as I can see from the code, this also applies to latest dev version.

Can anyone confirm this behaviour? I opened an issue request with a proposed fix for the API side of things https://github.com/processwire/processwire-issues/issues/1680

[wbmnfktr]

If you only test with this exact e-mail address the value is NOT valid and shouldn't pass the $sanitizer.
So if that's the case, the sanitizer works.

This shouldn't work: hans.müller@müller.com

This should work: hans.mueller@müller.com

[gebeer]

17 hours ago, wbmnfktr said:

If you only test with this exact e-mail address the value is NOT valid and shouldn't pass the $sanitizer.

That is not true anymore since 3.0.208. Docs for https://processwire.com/api/ref/sanitizer/email/ options say:

Quote

• allowIDN (bool|int): Allow internationalized domain names? (default=false) Specify int 2 to also allow UTF-8 in local-part of email [SMTPUTF8] (i.e. bøb).

I have verified that sanitizer accepts with the allowIDN option set to 2.

$sanitizer->email('hans.müller@müller.com', ['allowIDN' => 2];

But the occurences of $sanitizer->email() in InputfieldEmail.module and FieldtypeEmail.module need to be adjusted to allow saving of those values to the DB.

We are working on an app for an international corporation and need to import around 15.000 email addresses, some of which have characters like ä, ø etc. Their mailservers support these, so we have to support them also.

I'm totally not into reading RFCs, but doing some research I found that since RFC6532 (2012) internationalized email headers are a standard and even the local part of the email can be in UTF-8.

For anyone who wants to dig deeper, here are some links to get you started: 
https://stackoverflow.com/questions/69855149/email-with-special-characters-rejected-rfc-6532-and-quoted-printable 
https://github.com/roundcube/roundcubemail/issues/5120 
https://www.rfc-editor.org/rfc/rfc6532

[poljpocket]

I have drafted a new PR for this feature. The backend implementation is simple enough. The description strings for sure need some work.

See here: https://github.com/processwire/processwire/pull/259

It turns out, this is more complicated than anticipated because the browser validation fails.
In order to test the new feature, I have to change the <input>'s type to text or the browser won't allow sending the form.

[gebeer]

6 hours ago, poljpocket said:

It turns out, this is more complicated than anticipated because the browser validation fails.

It is odd that modern browsers do not support email validation following RFC 6530 out of the box. Popular PHP/JS libs like https://github.com/egulias/EmailValidator and https://www.npmjs.com/package/isemail/v/3.2.0 do support it.

Seems like we would have to come up with a custom email input JS validator to make this work in the GUI. That is certainly something that @ryan would have to implement. Not sure how other CMS/Frameworks support this feature. But would be awesome if ProcessWire did.

[wbmnfktr]

Wow... Didn't know that the $sanitizer and the specification do allow this now for the local-part as well.

?

[poljpocket]

@gebeer I have added another commit. I am not proud of that solution, but it works. I have changed the input type to text and added a simple pattern moving most of the validation burden to the server. This circumvents browsers' non-support for RFC6530 and uses the already-working implementation in $sanitizer.

This will never ever get added to core like this but at least this way, a custom fieldtype and inputfield could be extracted for your use case.

Let me know if that works for you.