Wiremail problem with charset and german umlauts

[Juergen]

Hello @ all,

today I have tried the PW wiremail function to send emails. Til now I have used phpmailer for all my emails without problems. Wiremail works and is sending emails but I have problems with German special characters (ü,ä,ö,...) and in the email body appears the content type and charset information.

The code that I use to send the mail:

                $mail = new wireMail();
                $mail->to($pages->get("template=imprint")->mail)->from($rfemail); // all calls can be chained
                $mail->subject($subjectcontactform); 
                $mail->bodyHTML($mailhtmlbody); 
                $mail->send(); 

This is the default code from Ryan for sending emails. Does someone has an idea what the cause could be for this behavior?

Best regards

[BitPoet]

I can see two issues there:

1. The Content-Transfer-Encoding needs to be 8bit to support raw utf8 characters
2. The subject needs to be encoded using either quoted-printable or base64 (quoted-printable is most common there, as it leaves parts of the subject readable)

Changing the affected functions like this in WireMail.php should get things working with umlauts:

	public function subject($subject) {
		//$this->mail['subject'] = $this->sanitizeHeader($subject); 	
		$this->mail['subject'] = '=?utf-8?Q?'.quoted_printable_encode($this->sanitizeHeader($subject)).'?='; 	
		return $this; 
	}

	public function ___send() {

		$header = '';
		$from = $this->from;
		if(!strlen($from)) $from = $this->wire('config')->adminEmail;
		if(!strlen($from)) $from = 'processwire@' . $this->wire('config')->httpHost; 

		$header = "From: " . ($this->fromName ? $this->bundleEmailAndName($from, $this->fromName) : $from);

		foreach($this->header as $key => $value) $header .= "\r\n$key: $value";

		$param = $this->wire('config')->phpMailAdditionalParameters;
		if(is_null($param)) $param = '';
		foreach($this->param as $value) $param .= " $value";		

		$header = trim($header); 
		$param = trim($param); 
		$body = '';
		$text = $this->body; 
		$html = $this->bodyHTML;

		if($this->bodyHTML) {
			if(!strlen($text)) $text = strip_tags($html); 
			$boundary = "==Multipart_Boundary_x" . md5(time()) . "x";
			$header .= "\r\nMIME-Version: 1.0";
			$header .= "\r\nContent-Type: multipart/alternative;\r\n  boundary=\"$boundary\"";
			$body = "This is a multi-part message in MIME format.\r\n\r\n" . 
				"--$boundary\r\n" . 
				"Content-Type: text/plain; charset=\"utf-8\"\r\n" . 
				//"Content-Transfer-Encoding: 7bit\r\n\r\n" . 
				"Content-Transfer-Encoding: 8bit\r\n\r\n" . 
				"$text\r\n\r\n" . 
				"--$boundary\r\n" . 
				"Content-Type: text/html; charset=\"utf-8\"\r\n" . 
				//"Content-Transfer-Encoding: 7bit\r\n\r\n" . 
				"Content-Transfer-Encoding: 8bit\r\n\r\n" . 
				"$html\r\n\r\n" . 
				"--$boundary--\r\n";
		} else {
			$header .= "\r\nContent-Type: text/plain; charset=\"utf-8\""; 
			$body = $text; 
		}

		$numSent = 0;
		foreach($this->to as $to) {
			$toName = $this->mail['toName'][$to]; 
			if($toName) $to = $this->bundleEmailAndName($to, $toName); // bundle to "User Name <user@example.com"
			if(@mail($to, $this->subject, $body, $header, $param)) $numSent++;
		}

		return $numSent; 
	}

 

If the names in From: or To: headers contain umlauts, they also need to be properly encoded. My fix for the subject() function above is a bit crude, as it doesn't care for long lines that should be wrapped. There's also the mb_encode_mimeheader function in PHP which does wrapping but doesn't care for word boundaries and might cause superfluous whitespaces in between words.

 

[Juergen]

Thanks @BitPoet

changing core files is not what I want. I will send my emails with PHPmailer again. It works really well without problems.

[adrian]

@Juergen - If you have a few moments, I think it would be helpful if you could test @BitPoet's changes to confirm they work so that these changes might be incorporated into the core. Thanks!

[Juergen]

Hello adrian,

I will test it tomorrow and post the result here.

Best regards

[Juergen]

Here is what I get after the changings from BitPoet:

As you can see the subject line at the top works well, but in the body area the German special characters will be replaced by letters and equal signs (for example "ü" will be replaced by "=C3=BC".

Also the content type, the charset and the content-transfer-encoding information will be displayed in the content area.

Best regards

[BitPoet]

That's strange, and it looks like there's something else (mailserver?) adding the quoted-printable encoding. But I noticed that I haven't based the code above on the most recent version of WireMail, so I'll run some tests myself and try to propose a solution that also takes cases like utf8 characters in names, line wrapping and such into account in an RFC-compatible way.

[BitPoet]

Issue #43 filed at GitHub and pull request submitted.

[hheyne]

Same problem here. I fixed it by using wiremailsmtp as addon to wiremail.

[Juergen]

After updating to PW 3.0.36 a mail body looks like this:

Problem is still there

[LostKobrakai]

The fact that the content type header and the transfer encoding is visible in the actual email body would suggest, that the email isn't parsed correctly in the first place. Without parsing the encoding type it's expected that decoding doesn't happen correctly. Any chance you could provide the emails source code and what email app you're using there?

[Juergen]

Email code:

$mailhtmlbody .= '<html><body><p><b>' . _t('Subject', 'Form') . ':</b> ' . $subjectcontactform . '</p><p><b>' . _t('Appointment for following events', 'Events') . ':</b></p><ul>' . $eventlisting . '</ul><p><b>' . ($furtherpersoncount + 1) . ' ' . $furtherpersonpost . '</b></p><ul><li>' . $firstname . ' ' . $lastname . ' (' . _t('Person who has registered for this event', 'Events') . ')</li>' . $allpersons . '</ul>' . $message . '<p><b>' . $senderinfo . '</b><br />' . $gendername . ' ' . $firstname . ' ' . $lastname . '</p><p><b>' . _t('Contact information', 'Form') . '</b><br />' . $user->fields->get('email')->$label . ': ' . $rfemail . '<br />' . _t('Phone/Mobile', 'Form') . ': ' . $phone . '<br /><br />IP: ' . $userip . '<br />' . _t('Date/time', 'Generic') . ': ' . $sendingtime . '</p></body></html>';

$mail = new wireMail();
$mail->to($pages->get("template=imprint")->mail)->from($rfemail); // all calls can be chained
$mail->subject($subjectcontactform); 
$mail->bodyHTML($mailhtmlbody); 
$mail->send(); 

The email will be sent from my standard provider, where I host all my websites.

[LostKobrakai]

I wasn't asking about how you send it, but what you actually receive and in which email program you're trying read it. If you could provide the source code of one of those emails we can determine if the sent email is even delivered in the correct state.

[Juergen]

It is office outlook , but unfortunately I am not able to view the sourcecode (its maybe not a HTML, but I have sent it as HTML-Mail). Show source code is disabled in this case. At the emails which were sent with PHPMailer I can view the the source.

[LostKobrakai]

An email's source code has nothing to do with the message being html or not html. Html might be used in the message, but that's about it. In outlook (at least for mac) I can do right click on an email and choose "open source" ("Quelle anzeigen" in german) to open the raw email in a texteditor. 

[Juergen]

I mean that the email is HTML not only the message .

Yes, but this is disabled in the mail - I have tried it. After searching the web I found the info, that this function is only possible if the email is HTML.

Quote

Einen Quelltext besitzt nur eine HTML-Mail und daher fehlt diese Option bei den beiden anderen Textformaten.

 

[LostKobrakai]

Not in the email, but rather in the list:

[Juergen]

The problem is that I have not this option the overview list.

Maybe I must add some settings to outlook

[LostKobrakai]

Can also be the difference of mac and windows or version difference. Microsoft is really not great at keeping things in sync :D. 

[LostKobrakai]

bernhard did send me emails to my microsoft office account and private one and to be honest I can't seem to get a hold on what's wrong with it. In one email app the mail is parsed as some kind of unknown file, even though the source does look quite correct. In Outlook (opened via outlook) the source does seem to miss any multipart bountries, and I'm not sure why. Opening the same mail in my other email app (but same account) does show the multipart boundries. It's really strange.

 

 

[adrian]

 

On 10/12/2016 at 11:52 AM, hheyne said:

Same problem here. I fixed it by using wiremailsmtp as addon to wiremail.

Might be good to also check @teppo's swift module as well to see if it also handles things properly. Of course the core wireMail should also work properly, would be good to know the current state of what does and doesn't work.

[dotnetic]

@BitPoet´s fix for quoted-printable mails is merged into dev. I updated my WireMail class and tried sending an E-Mail with the new class.

Now the HTML Part of the message with umlauts in it displays fine, but the title with german umlauts in it still does not display correct.
I tried all sorts of conversions and adding headers to the mail but nothing worked.

EDIT: This happens with multipart messages because they have no header like

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

But if I add those headers the rest of the mail is unreadable.

How do I get correct umlauts in the title? Edit: I found the solution, attached it to the end of my post.

Here is the some of the code I used:

$betr = 'öäüß testmail'
$mail = new \ProcessWire\WireMail();
//        $mail->header('Content-Transfer-Encoding','base64');
//        $mail->header('Content-Type', 'multipart/alternative; charset="utf-8"');
//        $mail->header('charset', 'utf-8');
        $mail->to('info@jensmartsch.de');
        $mail->from('info@jensmartsch.de);
        $mail->fromName("Jens Martsch");
//        $mail->subject(("$betr"));
        $mail->subject ('=?UTF-8?B?' . base64_encode($betr) . '?=');

//        $mail->subject(base64_encode('$betr'));
//        $mail->subject = "?utf-8?Q?" . quoted_printable_encode("öäüß The ï, ö, ë, ä, and é work, but when adding the ü it doesn't") . "?=";
        $mail->bodyHTML(($text));
        if (!$mail->send()){
            echo 'Fehler';
        }

This is the E-Mail header that I receive if I am using

$mail->subject($betr);

To: info@jensmartsch.de
Subject: =?utf-8?Q?=C3=B6=C3=A4=C3=BC=C3=9F Bewerbung Berechnungsingenieur / Messtechniker Aut=  omotive (m/w)?=
X-PHP-Originating-Script: 0:WireMail.php
From: =?utf-8?Q?Jens Martsch?= <info@jensmartsch.de>
X-Mailer: ProcessWire/WireMail
MIME-Version: 1.0
Content-Type: multipart/alternative;

What I see as the title in my mail programm is exactly the same as the Subject

SOLUTION: It seems that the length of my title was the problem. Now I try to find a solution for this as well. Maybe I just shorten the title.

Edited October 17, 2016 by jmartsch
Found the solution

[Juergen]

Strange! In my case it is the opposite: Subject is o.k. but body contains the letters numbers and equalsigns.

[dotnetic]

3 minutes ago, Juergen said:

Strange! In my case it is the opposite: Subject is o.k. but body contains the letters numbers and equalsigns.

Do you use the WireMail Class from dev Repository with bitpoets fixes?

My problem happens only because it is a multipart message and not a plain text message. I added this to my post.

[BitPoet]

@jmartsch: is your source file that sends the mail definitely saved as UTF-8?

I'm just doing some really extensive testing (including running mails through different sorts of mail servers along the way). I've spotted one issue with the position of quotes in the from: and to: fields but that is only triggered when there's a comma in the name, and there's a slight problem with HTML mails with attachments which should have nested parts since text + html should always have a multipart-alternative parent for the HTML to get displayed. I couldn't reproduce your or @Juergen's issues yet, though.