WireMailgun - An Alternative

[nbcommunication]

Hello. I'd somewhat rewritten WireMailMailgun a while back, to implement some more features and implement PW coding guidelines etc. My version has been somewhat in limbo, although there's been various discussions in the thread linked above. I've decided to release my version as a separate module called WireMailgun, as it has breaking changes and a slightly different implementation. I've also got it using the email validation v4 endpoint.

For simple usage, either module will do what you need it to do. If you need to do some more advanced things e.g inline images or adding data, this module will help you with that. Here's the readme...

WireMail Mailgun

Extends WireMail to use the Mailgun API for sending emails.

Installation

1. Download the zip file at Github or clone the repo into your site/modules directory.
2. If you downloaded the zip file, extract it in your sites/modules directory.
3. In your admin, go to Modules > Refresh, then Modules > New, then click on the Install button for this module.

API

Prior to using this module, you must set up a domain in your Mailgun account to create an API key. Add the API key and domain to the module's settings.

Usage

Usage is similar to the basic WireMail implementation, although a few extra options are available. Please refer to the WireMail documentation for full instructions on using WireMail, and to the examples below.

Extra Methods

The following are extra methods implemented by this module:

Chainable

The following methods can be used in a chained statement:

cc(string|array|null $email) - Set a "cc" email address.

• Only used when $batchMode is set to false.
• Please refer to WireMail::to() for more information on how to use this method.

bcc(string|array|null $email) - Set a "bcc" email address.

• Only used when $batchMode is set to false.
• Please refer to WireMail::to() for more information on how to use this method.

addData(string $key, string $value) - Add custom data to the email.

• See https://documentation.mailgun.com/en/latest/user_manual.html#attaching-data-to-messages for more information.

addInlineImage(string $file, string $filename) - Add an inline image for referencing in HTML.

• Reference using "cid:" e.g. <img src='cid:filename.ext'>
• Requires curl_file_create() (PHP >= 5.5.0)
• See https://documentation.mailgun.com/en/latest/user_manual.html#sending-inline-images for more information.

addRecipientVariables(array $recipients) - Add recipient variables.

• $recipients should be an array of data, keyed by the recipient email address
• See https://documentation.mailgun.com/en/latest/user_manual.html#batch-sending for more information.

addTag(string $tag) - Add a tag to the email.

• Only ASCII allowed
• Maximum length of 128 characters
• There is a maximum number of 3 tags allowed per email.

addTags(array $tags) - Add tags in a batch.

setApiKey(string $apiKey) - Override the Mailgun API Key module setting.

setBatchMode(bool $batchMode) - Enables or disables batch mode.

• This is off by default, meaning that a single email is sent with each recipient seeing the other recipients
• If this is on, any email addresses set by cc() and bcc() will be ignored
• Mailgun has a maximum hard limit of recipients allowed per batch of 1,000. Read more about batch sending.

setDeliveryTime(int $time) - The (unix)time the email should be scheduled for.

setDomainName(string $domain) - Override the "Domain Name" module setting.

setRegion(string $region) - Override the "Region" module setting.

• Valid regions are "us" and "eu"
• Fails silently if an invalid region is passed

setSender(string $domain, string $key) - Set a different API sender than the default.

• The third argument is $region which is optional
• A shortcut for calling setDomainName(), setApiKey() and setRegion()

setTestMode(bool $testMode) - Override the "Test Mode" module setting.

setTrackOpens(bool $trackOpens) - Override "Track Message Opens" module setting on a per-email basis.

• Open tracking only works for emails with bodyHTML() set

setTrackClicks(bool $trackClicks) - Override "Track Message Clicks" module setting on a per-email basis.

• Click tracking only works for emails with bodyHTML() set

Other

send() - Send the email.

• Returns a positive number (indicating number of emails sent) or 0 on failure.

validateEmail(string $email) - Validates a single address using Mailgun's address validation service.

• Returns an associative array. To return the response as an object, set the second argument to false
• For more information on what this method returns, see Mailgun's documentation.

getHttpCode() - Get the API HTTP response code.

• A response code of 200 indicates a successful response

Examples

Basic Example

Send an email:

$mg = $mail->new();
$sent = $mg->to("user@domain.com")
	->from("you@company.com")
	->subject("Message Subject")
	->body("Message Body")
	->send();

Advanced Example

Send an email using all supported WireMail methods and extra methods implemented by WireMailgun:

$mg = $mail->new();

// WireMail methods
$mg->to([
		"user@domain.com" => "A User",
		"user2@domain.com" => "Another User",
	])
	->from("you@company.com", "Company Name")
	->replyTo("reply@company.com", "Company Name")
	->subject("Message Subject")
	->bodyHTML("<p>Message Body</p>") // A text version will be automatically created
	->header("key1", "value1")
	->headers(["key2" => "value2"])
	->attachment("/path/to/file.ext", "filename.ext");

// WireMailgun methods
$mg->cc("cc@domain.com")
	->bcc(["bcc@domain.com", "bcc2@domain.com"])
	->addData("key", "value") // Custom X-Mailgun-Variables data
	->addInlineImage("/path/to/file-inline.jpg", "filename-inline.jpg") // Add inline image
	->addTag("tag1") // Add a single tag
	->addTags(["tag2", "tag3"]) // Add tags in a batch
	->setBatchMode(false) // A single email will be sent, both "to" recipients shown
	->setDeliveryTime(time() + 3600) // The email will be delivered in an hour
	->setSender($domain, $key, "eu") // Use a different domain to send, this one in the EU region
	->setTestMode(true) // Mailgun won't actually send the email
	->setTrackOpens(false) // Disable tracking opens
	->setTrackClicks(false); // Disable tracking clicks

// Batch mode is set to false, so 1 returned if successful
$numSent = $mg->send();

echo "The email was " . ($numSent ? "" : "not ") . "sent.";

Validate an Email Address

$mg = $mail->new();
$response = $mg->validateEmail("user@domain.com", false);

if($mg->getHttpCode() == 200) {
	echo $response->result == "deliverable" ? "Valid" : "Not valid";
} else {
	echo "Could not validate";
}

I hope it is useful!

Cheers,

Chris

[adrian]

Hi Chris,

Thanks for your hard work on this!

I am about to implement this new version and will be looking to use the batch sending option but noticed the 1000 recipient limit. Then I discovered the MailGun's PHP-SDK handles this for you automatically: 
https://stackoverflow.com/questions/36063534/mailgun-api-batch-sending-vs-individual-calls
https://github.com/mailgun/mailgun-php/tree/master/src/Message

Any thoughts about implementing the SDK into this module to make batch sending a little easier?

Thanks,
Adrian

[nbcommunication]

Hi @adrian,

I did look at using the SDK for the module, but felt it was overkill for what was required - A more fully featured Mailgun API module could be built with it, but that's a job for someone with more experience and need for it.

I'll have a look at this in the next week though. Not sure I'll be able to test it fully, but it should just be a case of splitting the to array into batches of 1000 and sending each of these. 

Cheers,

Chris

 

[nbcommunication]

Hi @adrian,

Got a chance to look at this today. Batch requests (batch mode on, to emails > 1000) now get split up into separate API requests with 1000 emails per request. 

The only way I could test this was to reduce the limit to 2 and then try sending a batch request to more than two email addresses. Seems to work fine, however I'd recommend enabling test mode and testing with this to confirm that it works correctly for emails being sent to 1000+ users prior to sending for real.

I've also moved the batchMode setting into the module config, so it can be set to ON by default.

Cheers,

Chris

[adrian]

@nbcommunication

Thanks Chris for the speedy work on this - sounds great!

The catch I am seeing is that if you try to send to less recipients than "batchLimit", the "if($deferredCount)" check fails and you end up without the "to" address being set. Can you confirm the issue at your end as well?

[nbcommunication]

Hi @adrian,

I'll check this later, but I'm pretty sure it should work as expected - "to" is set on line 469 regardless of batchMode, and then overridden if split into batches (line 594) so "to" and "recipientVariables" are limited to 1000 at a time.

Cheers,

Chris

[adrian]

2 minutes ago, nbcommunication said:

I'll check this later, but I'm pretty sure it should work as expected - "to" is set on line 469 regardless of batchMode, and then overridden if split into batches (line 594) so "to" and "recipientVariables" are limited to 1000 at a time.

I think the issue is that on line 469, the value of $this->mail["toName"] passed into getEmails() is an empty array. I think this is because we don't explicitly set it in batchMode when passing recipientVariables, so we need the logic inside the if($deferredCount) section to handle getting the email's details.

Thanks for looking into it.

[nbcommunication]

Hi @adrian,

How are the "to" email addresses being passed? As I understand it, the "toName" variable should be populated when calling WireMail::to()?

// WireMail.php 304-309

if(empty($toName)) $toName = $name; // use function arg if not overwritten
$toEmail = $this->sanitizeEmail($toEmail); 
if(strlen($toEmail)) {
	$this->mail['to'][$toEmail] = $toEmail;
	$this->mail['toName'][$toEmail] = $this->sanitizeHeaderValue($toName);
}

I think what you're saying is that you aren't using the to() method, and instead the "to" addresses are being set using addRecipientVariables()? I've tweaked the module to handle this implementation. 

Technically the recipients should be set using to() and addRecipientVariables() is for overriding the recipientVariables set by default from $mail["toName"], but I can't see the harm in allowing for this different implementation.

Cheers,

Chris

[adrian]

2 minutes ago, nbcommunication said:

I think what you're saying is that you aren't using the to() method, and instead the "to" addresses are being set using addRecipientVariables()?

There's a good chance I don't understand how this is meant to be done. I assumed that using addRecipientVariables() completely negated the need to use the to() method. In my mind, the to() method only makes sense if you are sending in non batch mode. How do you use it in batch mode for sending to multiple recipients - I don't understand that.

Also, I am curious about the toName() method. I thought the way you had things set up, in batch mode you should use this, but rather add a toName key to the recipients array.

This is basically what I am doing where $recipients is a PW page array:

    $recipientsArr = array();
    foreach($recipients as $u) {
        if($u->$email) {
            $recipientsArr[$u->$email] = array(
                'id' => $u->id,
                'toName' => $u->first_name . ' ' . $u->last_name,
            );
        }
    }

	$mailer = $mail->new();
	$mailer->setBatchMode(true);
	$mailer->addRecipientVariables($recipientsArr);
	$mailer->from('me@google.com', 'My Name);
	$mailer->subject($newsletter->title);
	$mailer->bodyHTML($message);
	$numSent = $mailer->send();

Does this make sense to you and should this work with your most recent commit?

[adrian]

Just testing this new version (using my code above) - it now works for one recipient when batchLimit is set to 1000, but if you are sending to 2 or more (presumably anything up to 1000), only the first recipient is sent the email.

Sorry, I think it is actually working - looks like my email client is not showing one of the messages because perhaps it thinks it's a duplicate even though they are going to different addresses?

[nbcommunication]

Hi @adrian,

Ah bugger, I'll see if I can figure that out...

As I understand it, WireMail calls should always use to() to set the recipients. However, it does make sense to allow for recipients to be set from addRecipientVariables(), as this prevents repetition should you want to set custom variables (id and name are the ones inferred from the "to" recipients). Hopefully this example makes some sense of that:

$to = array();
$recipientsArr = array();

foreach($recipients as $u) {
	if($u->$email) {

		$name = $u->first_name . ' ' . $u->last_name;
		$to[$u->email] = $name;

		/*
		If addRecipientVariables() isn't used, the recipient variable is inferred as:
		$recipientsArr[$u->$email] = array(
			'id' => $i++, // index increment (%recipient.id% in bodyHTML)
			'name' => $name, // (%recipient.name% in bodyHTML)
		);
		*/

		$recipientsArr[$u->$email] = array(
			'id' => $u->id,
			'toName' => $name, // (%recipient.toName% in bodyHTML)
		);
	}
}

// Previous usage
$mailer = $mail->new();
$mailer->setBatchMode(true);
$mailer->to($to);
$mailer->addRecipientVariables($recipientsArr);
$mailer->from('me@google.com', 'My Name');
$mailer->subject($newsletter->title);
$mailer->bodyHTML($message);
$numSent = $mailer->send();

// Now simplified, no need for to()
$mailer = $mail->new();
$mailer->setBatchMode(true);
$mailer->addRecipientVariables($recipientsArr);
$mailer->from('me@google.com', 'My Name');
$mailer->subject($newsletter->title);
$mailer->bodyHTML($message);
$numSent = $mailer->send();

// After the next update, records in $recipientsArr above must contain a value for either "name" or "toName"
// for this to be set as the "to" recipient name, e.g. $mailer->to($email, $name);

Some more info about toName:

$mail["toName"] is set by the core WireMail.php to() function as seen in the example in my previous post. The reason I pull the recipient emails from $mail["toName"] is that this array is a key=>value store of email addresses and the name set for them (if set), whereas $mail["to"] just contains the email addresses.

In WireMail::send(), $mail["to"] is traversed and the name is set if it exists in $mail["toName"]:

// WireMail.php 611-619
foreach($this->to as $to) {
  $toName = isset($this->mail['toName'][$to]) ? $this->mail['toName'][$to] : '';
  if($toName) $to = $this->bundleEmailAndName($to, $toName); // bundle to "User Name <user@example.com>"
  if($param) {
      if(@mail($to, $subject, $body, $header, $param)) $numSent++;
  } else {
      if(@mail($to, $subject, $body, $header)) $numSent++;
  }
}

I suspect this is probably a result of an earlier implementation of WireMail that didn't have the toName option - I think it makes more sense to traverse $mail["toName"] in this module's implementation.

The WireMail::toName() method isn't actually used by this module directly - in core WireMail this sets the "name" for the last added "to" email address, so could be used like so:

$mg = $mail->new();
$mg->to("email@example.com")
	->toName("An Example Name")
	->subject("A Subject")
	->bodyHTML("<p>Test</p>")
	->send();

// Sends to "An Example Name <email@example.com>"
// An alternative to $mg->to("email@example.com", "An Example Name");

On another note, while reviewing the docs I see there's a line length limit on X-Mailgun-Recipient-Variables which I'll likely need to handle too. Will see what I can do with this.

Cheers,

Chris

[adrian]

Thanks Chris,

I had forgotten about the ability to pass an array to the wireMail to() method - I was thinking it only handled one address or address / name pairing. 

Perhaps given that, there isn't really a need to pull the email and toName from the recipientVariables array, although it is a nice shortcut still. Sorry if I sent you on a wild goose chase with that.

One minor point - do you know why in my email client (Mac Mail) there is a comma after the "to" address, even if I only send it to one recipient:

It's not super important, but it seems a bit weird and wasn't happening before I started using batch mode.

[nbcommunication]

Hi @adrian,

I've made another update to the module. I've removed the inferring of "id" if addRecipientVariables() isn't used, as it was confusing and didn't really work anyway...

I'm not getting the comma issue (also using Mac Mail), and multiple batches is working for me also - if it still persists for you after the latest update, please let me know and I'll debug further in the morning.

I've added an example to the README which should hopefully explain batch sending/recipient variables a little more:

// If using batch mode, the recipient variable "name" is inferred from the `to` addresses, e.g.
$mg = $mail->new();
$mg->to([
		"user@domain.com" => "A User",
		"user2@domain.com" => "Another User",
	])
	->setBatchMode(true)
	->subject("Message Subject")
	->bodyHTML("<p>Dear %recipient.name%,</p>")
	->send();

// to = A User <user@domain.com>, Another User <user2@domain.com>
// recipientVariables = {"user@domain.com": {"name": "A User"}, "user@domain.com": {"name": "Another User"}}
// bodyHTML[user@domain.com] = <p>Dear A User,</p>
// bodyHTML[user2@domain.com] = <p>Dear Another User,</p>

// Alternatively, you can omit the `to` recipients if setting `recipientVariables` explictly e.g.
$mg = $mail->new();
$mg->setBatchMode(true)
	->addRecipientVariables([
		"user@domain.com" => "A User", // "name" is inferred
		"user2@domain.com" => [
			"name" => "Another User",
			"customVar" => "A custom variable",
		],
	])
	->subject("Message Subject")
	->bodyHTML("<p>Dear %recipient.name%,</p><p>Custom: %recipient.customVar%!</p>")
	->send();

// to = A User <user@domain.com>, Another User <user2@domain.com>
// recipientVariables = {"user@domain.com": {"name": "A User"}, "user@domain.com": {"name": "Another User", "customVar": "A custom variable"}}
// bodyHTML[user@domain.com] = <p>Dear A User,</p><p>Custom: %recipient.customVar%!</p>
// bodyHTML[user2@domain.com] = <p>Dear Another User,</p><p>Custom: A custom variable!</p>
// %recipient.customVar% only prints for second email, so not a particularly useful example!

// You can also use `addRecipientVariables()` to extend/override the inferred `recipientVariables` e.g.
$mg = $mail->new();
$mg->to([
		"user@domain.com" => "A User",
		"user2@domain.com" => "Another User",
	])
	->addRecipientVariables([
		"user@domain.com" => [
			"title" => "A User (title)",
		],
		"user2@domain.com" => [
			"name" => "Another User (changed name)",
			"title" => "Another User (title)",
		],
	])
	->setBatchMode(true)
	->subject("Message Subject")
	->bodyHTML("<p>Dear %recipient.name%,</p><p>Title: %recipient.title%!</p>")
	->send();

// to = A User <user@domain.com>, Another User (changed name) <user2@domain.com>
// recipientVariables = {"user@domain.com": {"name": "A User", "title": "A User (title)"}, "user@domain.com": {"name": "Another User (changed name)", "title": "Another User (title)"}}
// bodyHTML[user@domain.com] = <p>Dear A User,</p><p>Title: A User (title)!</p>
// bodyHTML[user2@domain.com] = <p>Dear Another User (changed name),</p><p>Title: Another User (title)!</p>

Cheers,

Chris

[nbcommunication]

Haha, up to version 1.1.6 now. I realised that ProMailer probably needs to use batch mode, so I've added some checks in the module config to set batch mode to on by default if it is installed.

[adrian]

Thanks for all the great updates (and the new examples) Chris,

Latest version is looking good so far - I expect I'll be testing more again tomorrow and will let you know if I notice any issues.

Cheers!

[nbcommunication]

22 hours ago, adrian said:

Just testing this new version (using my code above) - it now works for one recipient when batchLimit is set to 1000, but if you are sending to 2 or more (presumably anything up to 1000), only the first recipient is sent the email.

Sorry, I think it is actually working - looks like my email client is not showing one of the messages because perhaps it thinks it's a duplicate even though they are going to different addresses?

Yep - that happens! Mac Mail was doing the same to me when testing last night.

[adrian]

@nbcommunication - I am getting this error:

"Code 400: The parameters passed to the API were invalid - 'to' parameter is not a valid address. please check documentation"

when using the addRecipientVariables without also defining the `to` array. Here is my code:

    $recipientsArr = array();
    foreach($recipients as $u) {
        $recipientsArr[$u->$email] = array(
            'id' => $u->id,
            'email' => $u->$email
        );
    }

    $mailer = $mail->new();
    $mailer->setBatchMode(true);
    $mailer->addRecipientVariables($recipientsArr);

Everything seems to be working ok, but it would be nice is that error wasn't being logged. Any ideas? Thanks.

[nbcommunication]

Hi @adrian, will be back in the office on Friday so will take a look then.

Happy New Year to all when it comes!

Chris

[nbcommunication]

On 12/29/2019 at 11:12 AM, adrian said:

@nbcommunication - I am getting this error:

"Code 400: The parameters passed to the API were invalid - 'to' parameter is not a valid address. please check documentation"

when using the addRecipientVariables without also defining the `to` array. Here is my code:

    $recipientsArr = array();
    foreach($recipients as $u) {
        $recipientsArr[$u->$email] = array(
            'id' => $u->id,
            'email' => $u->$email
        );
    }

    $mailer = $mail->new();
    $mailer->setBatchMode(true);
    $mailer->addRecipientVariables($recipientsArr);

Everything seems to be working ok, but it would be nice is that error wasn't being logged. Any ideas? Thanks.

Hi @adrian, was just about to test and noticed the extra $ on line 3&5 - I think it should be $u->email?

[adrian]

Just now, nbcommunication said:

Hi @adrian, was just about to test and noticed the extra $ on line 3&5 - I think it should be $u->email?

That actually is correct - the email field for some recipient groups is "email", but for others it's "title" - $email is defined further up. Sorry for the confusion though ?

[nbcommunication]

Ah ok, I'll see what I can find ?

[nbcommunication]

HI @adrian - I've had a good look at it but can't find the issue running the code with some test data. Getting a 200 response and no errors logged.

Are you able to see the "to" address in the Mailgun log for the request? If so, is there anything there that might indicate what is causing the "to" address to be invalid?

[adrian]

Sorry @nbcommunication - in the middle of some other things at the moment. I actually ended up setting the "to" array because we had a newsletter going out this morning, so I will need to go back to dev setup to test without it. I'll take a look later and get back to you.

I do have another question for you though. When sending, the returned value is the number of emails sent, which for this module is the returned value of the apiRequest() method. The weird thing is that in my testing with a few recipients, the count was always correct, but my client sent our first proper mailout today and she said it reported more than the number of people that we have in our list. It's only a list of 130 users at the moment, but it returned 154. Have you seen anything like this before? Do you think this is something I should ask via MailGun support? Thanks.

 

[nbcommunication]

Hi @adrian,

That's bizarre! I've just done a test using 200 test emails (string length > 998 [nearer 15000] so using folding for recipient variables) and it returned the correct number of emails sent.

I don't think it is one for Mailgun support - the number/count generated is 1 if batch mode is off, and if it is on, as it is in this case, it returns the recipient variables count. If to() is set, it adds this to the recipient variables too, but because the recipient variables array is keyed by email, this shouldn't increase unless there are different emails in the to and recipient variables arrays. In other words, the count is internal; it's not returned by the Mailgun API.

My first point of attack on the problem would be the Mailgun logs, followed by adding a few $this->log($data["recipient-variables"]); or $this->log(count(json_decode($data["recipient-variables"], 1))); calls in the module where appropriate to try and figure out where the problem is occurring.

I don't actually have access to Mailgun logs here, so will get a better picture on Monday - perhaps sending in test mode is skewing my tests!

Are you using this with ProMailer? Wondering whether the issue is integration with another module?

Cheers,

Chris

[adrian]

Just a quick thought until I have more time to test. Because I don't have the "name" of the recipients, I am setting up $to in a loop like this:

$to[] = $u->email;

instead of:

$to['email'] = 'My Name';

Do you think that having a numeric instead of associative array could be a problem?