New blog post: ProMailer now available

[ryan]

This week ProcessWire ProMailer has been released, plus we’ve got a nice upgrade in our community support forum, and more—

https://processwire.com/blog/posts/promailer-now-available/

[adrian]

Your link to Cerberus is incorrect: 

https://tedgoas.github.io/Ce vs https://tedgoas.github.io/Cerberus/

[diogo]

Great work with the forum! To make it perfect, just add these to the css ?

.uk-background-primary, .uk-section-primary {
    -webkit-font-smoothing: antialiased;
    -moz-osx-font-smoothing: grayscale;
}
#masthead-logo {
    transform: translateY(-2px);
}
.uk-breadcrumb {
    transform: translateY(-1px);
}

 

@ryan, you might also consider removing the antialiasing from the body of the PW website, and add it only on the blue background areas. Here is the reasoning:

http://usabilitypost.com/2012/11/05/stop-fixing-font-smoothing/
https://www.zachleat.com/web/font-smooth/

[bartelsmedia]

As Processwire seems to be quite successful in Germany, some GDPR considerations:

• Are the subscriber email addresses stored with encryption in the database? GDPR enforces to use "state of the art" methods to protect user data which includes encryption.
• Is there a email black list of users who do not want receive a newsletter under any circumstances. Germany is quite lunatically rigid regarding "spam" emails with penalties of around $5000 for each unwanted received email. Example: User receives newsletter, denies to ever have subscribed and asks for a ceise and desist letter. If you then send another letter to such user, it will be expensive.
• Does the module save the newsletter subscriptions with sufficient evidences of the subscription? Time stamp, IP address and header (of course also saved with encryption in the database)?

 

 

 

[teppo]

17 hours ago, bartelsmedia said:

Are the subscriber email addresses stored with encryption in the database? GDPR enforces to use "state of the art" methods to protect user data which includes encryption.

Just a quick note on this one: to my best understanding GDPR doesn't explicitly require encryption. Encryption is mentioned as a suitable option for data processors and controllers in order to secure user data, but it isn't strictly speaking enforced – and thus not encrypting data (such as email addresses) wouldn't automatically mean that the product can't be GDPR compliant.

[ryan]

Quote

Are the subscriber email addresses stored with encryption in the database? GDPR enforces to use "state of the art" methods to protect user data which includes encryption.

Thanks for your questions. I suspect that what they are really referring to with encryption has to do with using HTTPS, though I could be wrong. That's where encryption would matter most here. If they are instead referring to private storage of the data on a server/DB, then "user data" is probably too general a term. Things like SSNs or financial info would be confidential user data that you might want to encrypt with a user's password as the key, so that it would not be reversible except by the user. Things like an email address or website URLs probably not.  That's because they are already public identifiers used by the internet as a network, passing through perhaps dozens of other servers on their way to their destination, and stored in non-encrypted logs by both originating and recipient recipient server. So if one is using some kind of confidential data as part of their email address, they probably shouldn't use it for... email.

Storage of any data, whether confidential or not, is of course "protected" by the access control of the server. At the other end of the spectrum, if one can access data already, encryption doesn't add any protection if the means of reversing the encryption is located on the same server where the encryption occurs (as it would have to be to make use of it on a website). That's why I think that GDPR statement must be referring to HTTPS, since encrypting these things at the server side would require decrypting them at the server side, and thus wouldn't be very useful (false sense of security). 

Nevertheless, if it's something you want it do, it would be fairly simple to implement, and I'm happy to outline the hooks necessary to do it. The big drawback is that the data would no longer be searchable. But if one has the need for whatever reason, then yes, it's definitely possible 

Quote

Is there a email black list of users who do not want receive a newsletter under any circumstances. Germany is quite lunatically rigid regarding "spam" emails with penalties of around $5000 for each unwanted received email. Example: User receives newsletter, denies to ever have subscribed and asks for a ceise and desist letter. If you then send another letter to such user, it will be expensive.

A user can only receive a newsletter if they subscribed to it and also confirmed that they wanted to receive it by verifying their identity via email. So that could not be considered "unwanted" email since one has to specifically opt-in to receive it. However, if there are some kind of public blacklist services or lists that you want to use to prevent users from even attempting to opt-in to a newsletter, then I could definitely add support for them. I'm not aware of any at present, so if you are, please let me know and I can get a closer look. Though I'd think just the public existence of such a blacklist would probably invite unwanted email to those addresses, from outside the jurisdiction of the blacklist. 

ProMailer is not a tool for sending spam, and if that's someone's intention ProMailer is not the right tool for that. Regardless of tool, if someone can make false claims and subject someone else to $5k per email of penalties, then I would probably eliminate email as part of any business model in those jurisdictions. So if that's the case, email would be dead in my book and I'd move user communication to another medium like SMS or even traditional postal mail. 

Quote

Does the module save the newsletter subscriptions with sufficient evidences of the subscription? Time stamp, IP address and header (of course also saved with encryption in the database)?

You can only subscribe if you confirm it with your email address (double opt-in). So the only situation where you can subscribe is if you yourself confirm it from your email. You can test it out from here if you'd like. But if someone breaks into your email, then yes they could subscribe you. If that's a concern, then this is another situation where I would avoid email as a medium for communications. We do store the timestamp of both subscription request and confirmation. No IP addresses or information about the browser client are stored by default, but you can store the information on your own if you'd like to. 

 

[teppo]

37 minutes ago, ryan said:

At the other end of the spectrum, if one can access data already, encryption doesn't add any protection if the means of reversing the encryption is located on the same server where the encryption occurs (as it would have to be to make use of it on a website). That's why I think that GDPR statement must be referring to HTTPS, since encrypting these things at the server side would require decrypting them at the server side, and thus wouldn't be very useful (false sense of security).

There is actually one benefit: if data is stored encrypted and the key is stored separately, even if it's on the same server, someone gaining illegitimate access to just your database but not full access to the server would mean that said data could be considered much less useful to the offending party. At the very least it could take them a long time to figure out what it is that they actually got their hands on.

Although this predates GDPR, in opinion 03/2014 the Article 29 Working Party stated that breach affecting only encrypted data may also mean that the source of the breach isn't subject to typical breach notification procedures:

Quote

Nevertheless, if the confidentiality of the key is intact, the data are in principle unintelligible to any person who is not authorised, thus the breach is unlikely to adversely affect the data subject and therefore doesn't need to be notified to the data subject. 

Just something to consider. I'm personally still not convinced that storing emails encrypted makes a whole lot of sense on a typical website, but I'm also not a lawyer, or a specialist in data privacy for that matter. Still, if my business relied largely on storing personal data, I would probably consider taking every precaution in order to avoid a full-blown disaster ?

[ryan]

Teppo, good points. Though on the servers I work from at least, you'd need access to the server account before you could ever get into the DB. Though I know this isn't representative of all environments. In general, I think DB backups are where there's more potential need to protect things. Those might be downloaded to the developers computer and then become more independent of the server, where encryption could actually be a more valuable protection. Encrypting emails does seem pretty silly given their purpose, so it's not something I'd do unless I had no choice, but for people where that's the case the good news is that it would be very simple to do. 

[bartelsmedia]

5 hours ago, teppo said:

Just a quick note on this one: to my best understanding GDPR doesn't explicitly require encryption. Encryption is mentioned as a suitable option for data processors and controllers in order to secure user data, but it isn't strictly speaking enforced – and thus not encrypting data (such as email addresses) wouldn't automatically mean that the product can't be GDPR compliant.

Here you go: http://www.privacy-regulation.eu/en/article-32-security-of-processing-GDPR.htm

[bartelsmedia]

5 hours ago, ryan said:

Thanks for your questions. I suspect that what they are really referring to with encryption has to do with using HTTPS, though I could be wrong. That's where encryption would matter most here. If they are instead referring to private storage of the data on a server/DB, then "user data" is probably too general a term. Things like SSNs or financial info would be confidential user data that you might want to encrypt with a user's password as the key, so that it would not be reversible except by the user. Things like an email address or website URLs probably not.  That's because they are already public identifiers used by the internet as a network, passing through perhaps dozens of other servers on their way to their destination, and stored in non-encrypted logs by both originating and recipient recipient server. So if one is using some kind of confidential data as part of their email address, they probably shouldn't use it for... email.

Storage of any data, whether confidential or not, is of course "protected" by the access control of the server. At the other end of the spectrum, if one can access data already, encryption doesn't add any protection if the means of reversing the encryption is located on the same server where the encryption occurs (as it would have to be to make use of it on a website). That's why I think that GDPR statement must be referring to HTTPS, since encrypting these things at the server side would require decrypting them at the server side, and thus wouldn't be very useful (false sense of security). 

Nevertheless, if it's something you want it do, it would be fairly simple to implement, and I'm happy to outline the hooks necessary to do it. The big drawback is that the data would no longer be searchable. But if one has the need for whatever reason, then yes, it's definitely possible 

A user can only receive a newsletter if they subscribed to it and also confirmed that they wanted to receive it by verifying their identity via email. So that could not be considered "unwanted" email since one has to specifically opt-in to receive it. However, if there are some kind of public blacklist services or lists that you want to use to prevent users from even attempting to opt-in to a newsletter, then I could definitely add support for them. I'm not aware of any at present, so if you are, please let me know and I can get a closer look. Though I'd think just the public existence of such a blacklist would probably invite unwanted email to those addresses, from outside the jurisdiction of the blacklist. 

ProMailer is not a tool for sending spam, and if that's someone's intention ProMailer is not the right tool for that. Regardless of tool, if someone can make false claims and subject someone else to $5k per email of penalties, then I would probably eliminate email as part of any business model in those jurisdictions. So if that's the case, email would be dead in my book and I'd move user communication to another medium like SMS or even traditional postal mail. 

You can only subscribe if you confirm it with your email address (double opt-in). So the only situation where you can subscribe is if you yourself confirm it from your email. You can test it out from here if you'd like. But if someone breaks into your email, then yes they could subscribe you. If that's a concern, then this is another situation where I would avoid email as a medium for communications. We do store the timestamp of both subscription request and confirmation. No IP addresses or information about the browser client are stored by default, but you can store the information on your own if you'd like to. 

 

The GDPR asks generally to minimize gathering and storing person identifiable information and if required, apply state of the art protection. A database with encrypted email is state of the art and protects user data in case of theft of the database (or its backups).

[bartelsmedia]

5 hours ago, ryan said:

You can only subscribe if you confirm it with your email address (double opt-in). So the only situation where you can subscribe is if you yourself confirm it from your email. You can test it out from here if you'd like. But if someone breaks into your email, then yes they could subscribe you. If that's a concern, then this is another situation where I would avoid email as a medium for communications. We do store the timestamp of both subscription request and confirmation. No IP addresses or information about the browser client are stored by default, but you can store the information on your own if you'd like to.

Please consider it a preconfigured option. We know cases where users refuse to ackknowledge that they registered and then sued for spam. It really is insane and a big business for the law industry. It would help, if ProMailer can provide as much evidence as possible (while it is totally clear, that there is no bullet proof evidence). 

I feel, that there is quite a culture clash between USA and in particular Germany regarding data privacy.

You may run this through google translate: https://www.e-recht24.de/news/marketing-seo/7980-newsletter-a-co-double-opt-verfahren-in-muss-dokumentiert-werden.html

[teppo]

10 hours ago, bartelsmedia said:

Here you go: http://www.privacy-regulation.eu/en/article-32-security-of-processing-GDPR.htm

Thanks! The part in question would be this: "Taking into account the state of the art, the costs of implementation and the nature, scope, context and purposes of processing as well as the risk of varying likelihood and severity for the rights and freedoms of natural persons, the controller and the processor shall implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk, including inter alia as appropriate [...]"

GDPR doesn't state that the controller and the processor must encrypt all personal data, period. Loosely translated it says that, taking a lot of things (such as the cost, nature, scope, context, and purpose of data processing) they must take measures appropriate to the risk and its likelihood, and one of the measures that may be appropriate depending on a lot of factors would indeed be encryption. The truth to the matter is that unless there's a court case stating essentially that everything (including specifically email addresses) must be encrypted during storage, this can be interpreted in a multitude of ways.

Then again, I'm still not a lawyer, and am mainly relying on opinions on the Internet – so don't take my word for it, and please do take every precaution you feel would be appropriate to your case ?

[bartelsmedia]

Quote

The truth to the matter is that unless there's a court case stating essentially that everything (including specifically email addresses) must be encrypted during storage, this can be interpreted in a multitude of ways.

So true. Anyway, you will be asked by the court why you have not made use of such "reasonable measure" like free & easy-to-apply AES encryption in a court case. And a friendly smile will not save you here.

I am not a lawyer but have witnessed many court cases to see the risk. The topic may be difficult to understand for non German businesses who enjoy less strict privacy rules. On the other hand, it would be really benefit of ProMailer compared to other newsletter systems if it would proactively add data security.

--

Regarding subscriber black list: Here is a court case which states, that you must make sure, that you must not send an email to any email of the entire TLD or risk hefty penalties. http://www.gesetze-bayern.de/Content/Document/Y-300-Z-BECKRS-B-2017-N-102179

So, the black list must enable users to optionally block an entire domain belonging to an individual.

Example:

1. A malicious person subscribes to your newsletter.  Think of a competitor or his buddy.

2. He refuses to ever have subscribed and sends you a cease & desist letter because you "spammed him".

3. You have no sufficient evidence to prove the subscription, you then block the particular email and sign the letter which makes it a binding agreement.

4. The guy uses a different email address under the same domain. He then gets another newsletter from you which violates the agreement.

You have to pay $3000-5000 to him.

The success of Mailchimp & Co. may partly be, because companies just can't stand all the legal traps anymore.

ProMailer could provide a solution to this problem with quite easy measures.

[dotnetic]

@bartelsmedia The case you linked to is to prevent unsolicited advertising emails.

An opt-in email is NOT an advertising email, depending on the contents of the email. If it just says, "you are subscribed", or "please confirm your subscription", without links to products or special offers, or something else that has advertising character, sending it is allowed.

16 hours ago, bartelsmedia said:

The GDPR asks generally to minimize gathering and storing person identifiable information and if required, apply state of the art protection. A database with encrypted email is state of the art and protects user data in case of theft of the database (or its backups).

Like @teppo said, it is not required to store the emails in an encrypted format.

I am also no lawyer but had to inform myself a lot about this, because I needed the information for some of my customers.

[ryan]

If all of this is correct, I personally think that the stakes are too high for sending any kind of email in a place where anyone can accuse another of emailing them and subject them to thousands in penalties from a single email sent by accident. It seems like that creates a dark market for people to pursue receipt of email as a litigation model. It seems like sending any email at all is huge risk where one accidental email could bankrupt you (reminds me of the US healthcare system). Personally, I wouldn't have the ability to pay a lawyer to even respond to such a complaint, so would be inclined to play it safe and simply remove email as a communications method from my business entirely. I don't think you could safely run a software like this one (IP.Board) with those kinds of restrictions.

But as far as ProMailer goes, if people subject to those kinds of laws still want to use it, I do think I can support much of what's been mentioned so far. Actually I think it's a good opportunity for ProMailer as a product to provide answers for these kinds of needs, and would enjoy implementing solutions for them. But I consider everyone here my friends, and with the stakes being so high, I would prefer that friends dealing with such laws stay far away from any software or service involved in sending email. Sending email sounds like a death trap as dangerous as swimming with crocodiles. I understand some will take the risk anyway, so I'll do my best to make sure ProMailer has answers for these kinds of things.

The only one I'm really not wild about is encrypting email addresses, just because that would place major limitations on the ability to search subscribers, which might increase the risks in other ways. For instance, Mike sends you a C&D letter, but you can't find Mike in your lists in order to remove him, so he ends up getting another email, and BAM, Mike gets to take over your life savings. But I think I can provide hooks for those that want to do it anyway. 

In my development version, this morning I added an option to log IP addresses with subscribe request and confirmation logs. It's off by default, but can be enabled in the module settings. As I understand it logging of IP addresses is not legal in some places, so anything that has potential to record IPs I usually keep disabled as a default. But the option will be there for those that want to enable it. 

With regard to a blacklist, we've been talking about it in the ProMailer board and I'm currently thinking a blacklist might be better supported in the core WireMail rather than just in ProMailer. That way you could blacklist an email address for anything in PW that might send an email via WireMail, rather than just ProMailer. For instance, modules like LoginRegister use WireMail to confirm account creation, so a lower-level blacklist could affect that module or any others too, in addition to ProMailer. That way an errant confirmation email or password-reset email won't cause someone to lose their retirement savings. I understand one of these blacklists can be individual email addresses, or entire domains, but am wondering about scale: are blacklists usually fairly small, or might it be an existing published list with thousands of domains/emails?

[HerTha]

email is kaputt

just saying...

[bartelsmedia]

Quote

are blacklists usually fairly small, or might it be an existing published list with thousands of domains/emails?

I would say, rather small (max a few dozens to hundreds for bigger companies). YMMV

[HerTha]

got a copy of ProMailer anyway ?

[Robin S]

On 3/19/2019 at 5:14 AM, ryan said:

The only one I'm really not wild about is encrypting email addresses, just because that would place major limitations on the ability to search subscribers, which might increase the risks in other ways.

I don't know much about encryption, but if the email address is the entirety of the field value (i.e. there is a dedicated field that holds the email address and nothing but the email address) then couldn't you encrypt the search value and then match that encrypted value in the DB to find subscribers who have that exact email address? Of course this wouldn't be a solution if you wanted to match a part value (e.g. all subscribers with somedomain.com in their email address) but it would be something at least.

[LostKobrakai]

The domain could be saved separately and probably not even be considered PII. What would be lost by using encryption is any kind of db powered fuzzy search for email addresses.

[Schwab]

How do I inline the css of the HTML Email? Is there a best practise with ProMailer?

Other tools like Campagin Monitor or MailChimp use a css inliner.

[dotnetic]

@Schwab Right now I am using the PHP class Emogrifier to inline the styles.

It works as easy as 

$emogrifier = new \Pelago\Emogrifier();

$html = '<html><h1>Hello world!</h1></html>';
$css = 'h1 {font-size: 32px;}';

$emogrifier->setHtml($html);
$emogrifier->setCss($css);
$mergedHtml = $emogrifier->emogrify();

Well, $html has to come from your page, and deliver correct HTML for every mail client. You have to generate that first. There are some preconfigured templates out there or you could use tools like MJML to create a template yourself.

[elabx]

Interesting app to design emails:  

https://imperavi.com/revolvapp/

[dotnetic]

Useful tools:

Design email in browser for free: https://mosaico.io @elabx
Preview your email in different clients before sending: https://emailpreviewservices.com/

[tomasanjosbarao]

I love ProcessWire and I'm considering buying ProMailer for the company I work in. We send a newsletter to ~1500 adresses about three times a month. Do you think it's imperative to subscribe an SMTP service like Mailgun? As long as SPF, DKIM and DMARC are correctly set up, is it reliable to use just your own SMTP, from your shared hosting? Is there anyone here using ProMailer without an external service? Thank you for your advice.