[WIP] ProcessMailer - A module for managing and sending newsletters

[abdus]

A client of mine was asking for a solution to send newsletter mails to a list of subscribers. I looked around for a module, but couldn't find any. Then I saw a screenshot on this blog post about UIKit update, and decided to recreate it. Huge thanks to @ryan for the inspiration.

The module uses regular pages for building HTML content. These pages can be used to create a fallback link in emails (i.e. "Use this link if you can't view email properly"). During render it injects $page->mailerMode, which can be used to change page output between text and HTML, or to show a simplified, email only HTML output.

Screenshots:

Main screen is just a list of items.

Create page:

Module configuration page:

During installation, module creates a page under admin for storing items similar to FieldtypeRepeater.

It also creates some fields for storing mail info

 

Todo:

• More testing
• Sending in batches (with a script that runs in background and real-time progress log)

Plans:

• Integration with Mailchimp (for subscriber lists)
• Automation (as a separate module for creating email content pages)

I'm hoping to complete and release the module in the following days. I think these features should be enough for the beginning, but I'm open to suggestions.

[tpr]

Imo a newsletter module is very needed for PW so I hope your project will be successful.

Here is a site profile that I used earlier for newsletters. It's very basic but perhaps you can get some ideas, eg using Emogrifier, preview newsletter with the View link in a panel, etc.

https://github.com/rolandtoth/pwnl

Some screenshots here: 

 

[abdus]

Thanks @tpr , it looks quite useful, I'll be sure to check it out.

[bernhard]

great to hear that you are working on that, indeed very needed! how do you plan to send the emails? imho it would be nice to be able to easily integrate services like https://www.sparkpost.com or https://sendy.co/ . i'm not sure but i think for large amounts of emails it is better to use some kind of substitution based sending method (https://developers.sparkpost.com/api/substitutions-reference.html ).

good luck for this module and have fun

[maxf5]

awesome project! an additional "newsletter subscribe" addon for the frontend would be awesome and a listing for the subscribers in the backend. So alternatively you can send the newsletter to these recipients instead of a txt file.

[abdus]

  On 10/11/2017 at 8:49 AM, bernhard said:

it would be nice to be able to easily integrate services like https://www.sparkpost.com or https://sendy.co/

Expand  

I'm thinking of integrating mailing services like Mailgun, SparkPost, SendGrid (currently working on Mailgun), and their `Mailing List` features via their APIs, because it's much easier to delegate potentially difficult points to a service experienced in this kind of work, instead of using regular SMTP to send emails (not to say I'm discarding SMTP option)

  On 10/14/2017 at 12:13 PM, maxf5 said:

"newsletter subscribe" addon for the frontend

Expand  

Using a mailing service also brings an easier way to add/remove subscribers to a list. With a method like $modules->ProcessMailer->getSubscriptionForm($options), I can return an instance of InputfieldForm, and developers can modify and render it as they see fit. Then I can issue a call to their API and add subscriber to a list.

[szabesz]

Great news @abdus! I think the ProcessWire community is really missing a full featured newsletter module, that is a module which supports out of the box (but hookable and modifiable) subscription forms, subscriber listings and sending emails via SMPT and popular service providers. Sounds like you are on the right track   

[abdus]

I've implemented subscriber list feature for Sendgrid, Mailgun and Sparkpost. One small complication with Sparkpost is that it doesn't really support addition of individual subscribers. It requires posting a complete list of subscribers with each update, which may not be ideal for large lists.

  Quote

If a recipients array is provided in the update request, it must contain the complete recipient list and all relevant recipient fields whether they are being changed or not. The new recipients will completely replace the existing recipients. The number of accepted recipients and the number of rejected recipients will only be returned if a recipients array is provided in the request.

Expand  

For extensibility, I've created a base module and each service extends and overrides methods for creating and handling subscription and list forms, API calls etc, this is because each service has different requirements and parameters for every operation.

This is the subscriber list form for Mailgun

For SparkPost:

and for SendGrid:

 

This is how subscribers lists view look

 

[abdus]

I'm also open to suggestions for other services. I didn't really like how SparkPost API handles subscriber lists, SendGrid is manageable but its terminology is very convoluted. I'll develop a service module for MailChimp as well, but haven't had the time to do anything about it yet.

[dragan]

  On 10/10/2017 at 3:44 PM, abdus said:

Sending in batches (with a script that runs in background and real-time progress log)

Expand  

Perhaps SwiftMailer would be a good library (also, apparently, it can detect rejected addresses)

https://swiftmailer.symfony.com/docs/sending.html#sending-emails-in-batch

[abdus]

I've added Mailchimp support

A few more screenshots:

Subscription Lists - Services

New Subscription List - Mailchimp

Module Configuration - Service Selection

[abdus]

Guys, I need some guidance.

I'm trying to come up with a good strategy for creating subscription forms. But it feels like I'm complicating this too much.

To give you some background:

• Multiple services be used simultaneously.
  For example: Mailgun for internal conversations, Mailchimp for promotional emails.
• A service can have multiple subscription lists.
  For example: one for weekly news, one for offers.
• A list can have only one subscription form for adding new members.
  It doesn't make sense to have multiple forms with different set of fields, which may result in empty columns for a subset of member records.
  At the same time, not all fields are necessary, but email must be specified at minimum.
• A form cannot expose anything to the frontend about the backend implementation to prevent spamming.
  For example Mailgun identifies lists using mail addresses (e.g. newsletter@example.com) that can be used to send mail to all members of the list.
  This means I have to assign unique IDs for subscription forms, then inject ID into each form as a hidden field.

What I want to offer:

• Form customization.
  Developer should be able to add/remove fields to/from a subscription form. I can build a minimal form using required fields only, and let developer extend it by adding new fields and hook into relevant methods of service class

Such as:

function ___buildSubscriptionFrom($list) {}
function ___addMemberToList($memberData, $list) {}

$this->addHookAfter('MailerServiceMailgun::buildSubscriptionFrom', function($e) {
    $list = $e->arguments(0);
    if ($list != 'newsletter@example.com') return;

    $form = $e->return;
    $e->add([
        'type' => 'text',
        'label' => 'Your age',
        'name' => 'age'
    ]);
});

$this->addHookBefore('MailerServiceMailgun::addMemberToList', function($e) {
    $data = $e->arguments(0);
    $data['vars'] = ['age' => $e->input->post->age];
    $e->setArgument(0, $data);
});

Another way of offering customization is by preparing a form for creating subscription forms and let users with certain permissions (mailer-forms-admin) pick and rename whichever fields he/she wants to use.

This option would open up the possibility of developing a Textformatter in the future for quickly embedding forms to any field content. Such as:

[[mailer-subscribe list=blog-list]]

Which would display a subscription form for subscribing to blog updates.

Now, writing this, providing hooks seemed like a good plan which I will put to use now, but I'm open to ideas.
  

[flydev]

  On 10/16/2017 at 2:40 PM, abdus said:

I'm trying to come up with a good strategy for creating subscription forms. But it feels like I'm complicating this too much.

...but I'm open to ideas

Expand  

I personally have the following module running to handle users to subscribe and unsubscribe to a newsletter. If your module could take into account @justb3a's module, it would be awesome. It already render the subscription form and create the proper user with the right role on submission.

 

 

[horst]

I don't know how the mailer services are get integrated into this new and very welcomed module, @abdus. One thing I would like to see is a modulare architecture, especially for the mailer services. They should be available as wiremail-modules, and the newsletter creator should relay on these modules, already implemented into pw.
I also second @flydevto integrate @justb3a s module. We should try to make our modules modulare, that we are able to focus on our main cases and don't have to be redundant.

[abdus]

  On 10/17/2017 at 5:54 AM, flydev said:

If your module could take into account @justb3a's module, it would be awesome.

Expand  

So far I haven't considered subscribing new users by creating a native user. I've integrated 4 services (MailChimp, Mailgun, Sendgrid, SparkPost) that allows subscription lists (recipient list/contacts/mailing lists,,, each with its own set of terminology, ugh) and a new subscriber is added to these lists. I think using these services in conjunction with @justb3a's could be better, in that I can hook into NewsletterSubscribe module and add the new user with `newsletter` roles to an one of the lists. Or simply, I could add an option to monitor user creation and add/remove when that user to a list if the user matches a certain selector.

However, though very unlikely, there could be some issues with networking / API reliability and sending a request to add/remove the subscriber may fail. Not sure how I can deal with syncing issues, except maybe retrying it LazyCron etc.

  On 10/17/2017 at 7:54 AM, horst said:

I don't know how the mailer services are get integrated into this new and very welcomed module, @abdus. One thing I would like to see is a modulare architecture, especially for the mailer services. They should be available as wiremail-modules, and the newsletter creator should relay on these modules, already implemented into pw.

Expand  

Mailer services are individual modules extending a base module so that I can combine all under ProcessMailer. But, they all can be used in isolation (with public methods, hooks etc).

As for each being WireMail modules, I realized yesterday that to send emails via these services, service modules have to extend WireMail to some degree, so that's what I'll go with. But it would also mean I'd have degrade my base module into an interface and duplicate a good chunk of code. This would remove majority of the coupling between ProcessMailer and its services to the degree that I could release them one by one, rather than all packed under ProcessMailer. But it will postpone the release by at least a few weeks.

Another option is that I could allow injection of a WireMail object as a parameter into one of the methods and modify the objects to suit service APIs, instead of converting them into individual WireMail modules.

<?php namespace ProcessWire;

public function ___sendEmail(WireMail $mail) {
    // modify $mail or extract relevant info and post it to mailer service
    $to = $mail->to();
    $from = $mail->from();
    // ...

    $this->send($apiEndpoint, $data);
}

// ===========

$mail = new WireMail();
$mail->to(...)
    ->from(...)
    ->subject(...)
    ->bodyHTML(...);
$list = 'newsletter@example.com';
$modules->MailerServiceMailgun->sendEmailToList($mail, $list);

 

[szabesz]

  On 10/17/2017 at 9:26 AM, abdus said:

So far I haven't considered subscribing new users by creating a native user.

Expand  

Your module should not do it, especially if you can make it play together well with Newsletter Subscription module.

  On 10/17/2017 at 9:26 AM, abdus said:

However, though very unlikely, there could be some issues with networking / API reliability and sending a request to add/remove the subscriber may fail.

Expand  

Because it is very unlikely, I think you can deal with this issue in another version in the future. No need to solve "all possible" issues in the first few releases  

  On 10/17/2017 at 9:26 AM, abdus said:

Another option is that I could allow injection of a WireMail object as a parameter into one of the methods and modify the objects to suit service APIs, instead of converting them into individual WireMail modules.

Expand  

+1  

[tpr]

Are you plan to integrate some statistics into it (clicked links, opened, etc)? 
(or leave it to third party services, not sure they support it)

[abdus]

  On 11/4/2017 at 2:37 PM, tpr said:

Are you plan to integrate some statistics into it (clicked links, opened, etc)? 
(or leave it to third party services, not sure they support it)

Expand  

That would be up to 3rd party services (almost all services support some type of tracking) or developer (by creating trackable links, 1x1px tracking gifs etc)

[adrianromega]

Hello

It would like to play with this module, is there any link available to download and try it on?

Thanks.

[abdus]

  On 11/8/2017 at 2:34 PM, adrianromega said:

Hello

It would like to play with this module, is there any link available to download and try it on?

Thanks.

Expand  

Not yet, I'll update the post once I get the alpha version ready.

[Lenz]

Hello @abdus, congrats on your really great venture. Much appreciated.

Is there any news about your newsletter module?

[LukeCage]

  On 11/8/2017 at 4:16 PM, abdus said:

Not yet, I'll update the post once I get the alpha version ready.

Expand  

Is there any news for this plugin? :)

[gizomo]

Hello @abdus.

Do you have a progress with your ProcessMailer module?

[netcarver]

@gizomo Here are a few other options for your consideration, in case this is not ready for sharing. In no particular order (and I haven't used any yet - so no recommendations)...

• https://github.com/nicoknoll/ProcessNewsletter
• https://github.com/pmarki/ProcessSendNewsletter
• https://github.com/rolandtoth/pwnl
• https://github.com/justb3a/processwire-newslettersubscription
• https://github.com/dauni/processwire-newslettermanagement

Hope that helps.

</redirect>

[gizomo]

Thanks, @netcarver