PipeEmailToPage - proposed new module

[MarkE]

I am looking at making a replacement/alternative module for ProcessEmailToPage for 2 reasons:

1. I want something where a webmaster can set up the process without needing access to cPanel to set up new email addresses.
2. The ProcessEmailToPage module is not really supported any more because it relies on Flourish, which is 12 years old and is no longer maintained.

I'm sure @Petedoesn't mind!

The main difference is that PipeEmailToPage will be a 'push' rather than a 'pull' approach. In cPanel, you will set up a pipe to a php file for 'Default addresses' (so they do not have to be specifically defined). The php file will be something like /site/modules/PipeEmailToPage/emailpipe.php. Thanks to some help from @BitPoet, I have the pipe working. There are a few gotchas to watch for, like files with CRLF not LF, and you need to set the emailpipe.php to execute permission 755, which it would not normally have. Then there will be a module script to handle various settings and maybe a Process page to view emails.

My next step is to add an email parser. There are a number out there. In particular https://github.com/zbateson/mail-mime-parser and https://github.com/php-mime-mail-parser/php-mime-mail-parser, but maybe others too. Each of those installs via composer; however, I'm not sure I want that, as all the code for the module should be packaged within it, I think, rather than rely on external packages. Also the second of those two requires php extensions that may not be installed (but are probably not difficult to add).

I would appreciate any thoughts on (a) php email parsers and (b) the use of composer vs including the code in the module. Or anything else relavant to this project 🙂.

[BitPoet]

It's always a tricky decision whether to include packages or have the user run composer. If license terms allow it, I've so far included the full source to make sure users don't end up with incompatibilities introduced by future versions. This, of course, sometimes means to react quickly if a vulnerability is discovered in a package my module relies upon, but I still prefer that to dysfunctional installations.

About using parser libraries at all - PHPs native capabilities are a PITA, and there are reasons why almost everybody uses a third party parser package (I had to rely on native stuff for a bunch of email parsing scripts that extracted contents of non standard conformant emails, and that was no fun at all).

Which to choose? A hard decision. Both are mature projects. One thing that comes to mind would be the dependency on PECL mailparse, which also has a known memory leak that can hit hard if one tries to process a larger number of mails in a loop. This and more intuitive handling of addresses could be a point towards Mail-Mime-Parser. On the other hand, a few content part handling methods feel more intuitive for PHP-Mime-Mail-Parser. No clear winner there, but I'd personally go for the simpler option from the user's perspective (i.e. Mail-Mime-Parser).

You might consider working with a queue directory as an option instead of using a direct pipe, which would improve resilience and performance. In cases like the database going away, all emails could still be written to the directory and processed later in a (lazy) cron job. Interfacing software is my main job description, and one philosophy I've acquired is to always finish pipe jobs as fast as possible.

[MarkE]

Thanks @BitPoet. That’s really helpful. I’ve pretty much decided on Mime-Mail-Parser for the reasons you state. So far I’ve installed it with composer, but will reconsider that. All the basics are now working nicely 😀. This includes a hookable method to allow people to add custom processing to check sender credentials. I will add whitelists and blacklists and am looking at additional anti-spam techniques. I also have quite a bit of work to do on the module config and Process page. 
Re queuing, I had considered that, but ditched it in favour of more rapid immediate processing. However, the point you raise about database availability is a good one as it might cause emails to just get lost. Maybe overcomplicating things, but how about a try-catch, just queuing it if there is a failure? This would need to be carefully crafted, as any error messages would result in the pipe failing. The advantage of always queuing is that it avoids any risk of errors being generated from attempting to access PW. I guess a 1 minute delay is acceptable but the instantaneous processing has been helpful while debugging. 

[BitPoet]

7 hours ago, MarkE said:

how about a try-catch, just queuing it if there is a failure

The problem is that PHP has a bunch of errors that can't be caught (or better worded, not worked around with custom error handlers - try-catch only catches Exceptions, which are just one side of the medal). The number of unrecoverable errors has been dwindling while version numbers rose, but there are still a few left, and they tend to show up unexpected.

Re queueing or not: you can always implement both. It could even come with its own script. If the filtering and page creation is handled by methods in your module, it should even be a a piece of cake. Are you planning to open source your module? If yes, I could implement the feature and send a pull. The thought behind that is that there are different ways to deliver mails and different reasons to add them as pages, from posting blog content to implementing a support ticket system on top of PW (an idea that has been nagging me for a long time), and probably a lot of others.

[MarkE]

36 minutes ago, BitPoet said:

The problem is that PHP has a bunch of errors that can't be caught

That's what I feared and that suppression might not fully work.

36 minutes ago, BitPoet said:

Re queueing or not: you can always implement both

Good idea. I'll look into that.

37 minutes ago, BitPoet said:

Are you planning to open source your module?

Definitely. I am trying to build it to be as general-purpose as possible, although my immediate need is to replace a hacked version of ProcessEmailToPage in a specific application. Hence the provision of hooks. Once I have the components in place, I will put it on GitHub. My view is that open source benefits everyone, including the original developer.

[MarkE]

1 hour ago, BitPoet said:

there are different ways to deliver mails and different reasons to add them as pages, from posting blog content to implementing a support ticket system on top of PW

My planned approach to this was to allow multiple templates as 'parents' for child mails. The mail will be saved under whichever parent has the matching 'to' address. It is then up to the user to use a LazyCron to process the mail pages as they wish (with different processing possible for different parent templates). So, for example, 'blog@mydomain.com' can be used to process to a blog whereas 'maillist@mydomain.com' can forward the mail to a list.

[BitPoet]

24 minutes ago, MarkE said:

My planned approach to this was to allow multiple templates as 'parents' for child mails. The mail will be saved under whichever parent has the matching 'to' address. It is then up to the user to use a LazyCron to process the mail pages as they wish (with different processing possible for different parent templates). So, for example, 'blog@mydomain.com' can be used to process to a blog whereas 'maillist@mydomain.com' can forward the mail to a list.

Nice. I didn't think of having "intermediate" mail pages for further processing, but it does make sense.