PipeEmailToPage - proposed new module

[MarkE]

I am looking at making a replacement/alternative module for ProcessEmailToPage for 2 reasons:

1. I want something where a webmaster can set up the process without needing access to cPanel to set up new email addresses.
2. The ProcessEmailToPage module is not really supported any more because it relies on Flourish, which is 12 years old and is no longer maintained.

I'm sure @Petedoesn't mind!

The main difference is that PipeEmailToPage will be a 'push' rather than a 'pull' approach. In cPanel, you will set up a pipe to a php file for 'Default addresses' (so they do not have to be specifically defined). The php file will be something like /site/modules/PipeEmailToPage/emailpipe.php. Thanks to some help from @BitPoet, I have the pipe working. There are a few gotchas to watch for, like files with CRLF not LF, and you need to set the emailpipe.php to execute permission 755, which it would not normally have. Then there will be a module script to handle various settings and maybe a Process page to view emails.

My next step is to add an email parser. There are a number out there. In particular https://github.com/zbateson/mail-mime-parser and https://github.com/php-mime-mail-parser/php-mime-mail-parser, but maybe others too. Each of those installs via composer; however, I'm not sure I want that, as all the code for the module should be packaged within it, I think, rather than rely on external packages. Also the second of those two requires php extensions that may not be installed (but are probably not difficult to add).

I would appreciate any thoughts on (a) php email parsers and (b) the use of composer vs including the code in the module. Or anything else relavant to this project 🙂.

[BitPoet]

It's always a tricky decision whether to include packages or have the user run composer. If license terms allow it, I've so far included the full source to make sure users don't end up with incompatibilities introduced by future versions. This, of course, sometimes means to react quickly if a vulnerability is discovered in a package my module relies upon, but I still prefer that to dysfunctional installations.

About using parser libraries at all - PHPs native capabilities are a PITA, and there are reasons why almost everybody uses a third party parser package (I had to rely on native stuff for a bunch of email parsing scripts that extracted contents of non standard conformant emails, and that was no fun at all).

Which to choose? A hard decision. Both are mature projects. One thing that comes to mind would be the dependency on PECL mailparse, which also has a known memory leak that can hit hard if one tries to process a larger number of mails in a loop. This and more intuitive handling of addresses could be a point towards Mail-Mime-Parser. On the other hand, a few content part handling methods feel more intuitive for PHP-Mime-Mail-Parser. No clear winner there, but I'd personally go for the simpler option from the user's perspective (i.e. Mail-Mime-Parser).

You might consider working with a queue directory as an option instead of using a direct pipe, which would improve resilience and performance. In cases like the database going away, all emails could still be written to the directory and processed later in a (lazy) cron job. Interfacing software is my main job description, and one philosophy I've acquired is to always finish pipe jobs as fast as possible.

[MarkE]

Thanks @BitPoet. That’s really helpful. I’ve pretty much decided on Mime-Mail-Parser for the reasons you state. So far I’ve installed it with composer, but will reconsider that. All the basics are now working nicely 😀. This includes a hookable method to allow people to add custom processing to check sender credentials. I will add whitelists and blacklists and am looking at additional anti-spam techniques. I also have quite a bit of work to do on the module config and Process page. 
Re queuing, I had considered that, but ditched it in favour of more rapid immediate processing. However, the point you raise about database availability is a good one as it might cause emails to just get lost. Maybe overcomplicating things, but how about a try-catch, just queuing it if there is a failure? This would need to be carefully crafted, as any error messages would result in the pipe failing. The advantage of always queuing is that it avoids any risk of errors being generated from attempting to access PW. I guess a 1 minute delay is acceptable but the instantaneous processing has been helpful while debugging.