MarkE

Module now released - see Interested in any feedback @BitPoet

Season's greetings ProcessWirers. My gift to you is a new module called PipeEmailToPage which you can get from GitHub here: https://github.com/MetaTunes/PipeEmailToPage/tree/main. Fairly 'alpha' at the moment, but I have been using it successfully on a live site for a week or so. I'll test it some more before releasing it to the modules library. I designed this as a replacement for ProcessEmailToPage, which relies on the flourish library which is no longer maintained. It also has a fundamentally different method of operation in that it is 'push' rather than 'pull'. The email addresses are created 'virtually' in the sense that they are not necessarily real email addresses but are held in pages whose templates are linked in the module config. The module works when emails are piped to a script (emailpipe.php) on the server which processes the email and creates the page. The pipe needs to be defined in your hosting service's cPanel or similar. If you define the pipe in the 'Default address' in cPanel, all emails sent to the domain will be processed unless they are specified separately. The 'to' addresses will be matched against the email addresses defined in 'parent' pages holding the virtual addresses. If you define the pipe for a specific email address, only emails sent to that address will be processed (but you can define the same pipe for multiple addresses). A major advantage of the module is that the email addresses can be defined entirely within PW. This means that they can be maintained by someone with no access to cPanel once the developer has set up the pipe. Further details are in the readme. I suggest you test thoroughly before using (and take careful note of the 'points to note' in the readme). Turn on some of the logging statements if you wish. Also, check your mail delivery in cPanel to look for errors there. If you report errors to me, I will do my best to hunt them, but please include as much info as possible. I will also be grateful for any suggested code improvements and/or PRs. 🎄

Apologies to anyone ( @BitPoet, @teppo ?) who may have downloaded the module - I left some code in by mistake which crashed it. Now hopefully fixed.

Also, it would be nice to fix the softaculous text to not say "ProcessWire is a free PHP5 content management system and framework ". That is a bit of a turn-off.

Hi @BitPoet and anyone else interested. An initial release of this module is available here: https://github.com/MetaTunes/PipeEmailToPage Still quite a lot of refinements to add (and no doubt some bugs to fix), including adding a queue as discussed above. Also you need to install the zbateson parser with composer at the moment. Any feedback is appreciated.

My planned approach to this was to allow multiple templates as 'parents' for child mails. The mail will be saved under whichever parent has the matching 'to' address. It is then up to the user to use a LazyCron to process the mail pages as they wish (with different processing possible for different parent templates). So, for example, 'blog@mydomain.com' can be used to process to a blog whereas 'maillist@mydomain.com' can forward the mail to a list.

That's what I feared and that suppression might not fully work. Good idea. I'll look into that. Definitely. I am trying to build it to be as general-purpose as possible, although my immediate need is to replace a hacked version of ProcessEmailToPage in a specific application. Hence the provision of hooks. Once I have the components in place, I will put it on GitHub. My view is that open source benefits everyone, including the original developer.

Thanks @BitPoet. That’s really helpful. I’ve pretty much decided on Mime-Mail-Parser for the reasons you state. So far I’ve installed it with composer, but will reconsider that. All the basics are now working nicely 😀. This includes a hookable method to allow people to add custom processing to check sender credentials. I will add whitelists and blacklists and am looking at additional anti-spam techniques. I also have quite a bit of work to do on the module config and Process page. Re queuing, I had considered that, but ditched it in favour of more rapid immediate processing. However, the point you raise about database availability is a good one as it might cause emails to just get lost. Maybe overcomplicating things, but how about a try-catch, just queuing it if there is a failure? This would need to be carefully crafted, as any error messages would result in the pipe failing. The advantage of always queuing is that it avoids any risk of errors being generated from attempting to access PW. I guess a 1 minute delay is acceptable but the instantaneous processing has been helpful while debugging.

I am looking at making a replacement/alternative module for ProcessEmailToPage for 2 reasons: I want something where a webmaster can set up the process without needing access to cPanel to set up new email addresses. The ProcessEmailToPage module is not really supported any more because it relies on Flourish, which is 12 years old and is no longer maintained. I'm sure @Petedoesn't mind! The main difference is that PipeEmailToPage will be a 'push' rather than a 'pull' approach. In cPanel, you will set up a pipe to a php file for 'Default addresses' (so they do not have to be specifically defined). The php file will be something like /site/modules/PipeEmailToPage/emailpipe.php. Thanks to some help from @BitPoet, I have the pipe working. There are a few gotchas to watch for, like files with CRLF not LF, and you need to set the emailpipe.php to execute permission 755, which it would not normally have. Then there will be a module script to handle various settings and maybe a Process page to view emails. My next step is to add an email parser. There are a number out there. In particular https://github.com/zbateson/mail-mime-parser and https://github.com/php-mime-mail-parser/php-mime-mail-parser, but maybe others too. Each of those installs via composer; however, I'm not sure I want that, as all the code for the module should be packaged within it, I think, rather than rely on external packages. Also the second of those two requires php extensions that may not be installed (but are probably not difficult to add). I would appreciate any thoughts on (a) php email parsers and (b) the use of composer vs including the code in the module. Or anything else relavant to this project 🙂.

This works: #!/usr/local/bin/php -q <?php namespace ProcessWire; include(__DIR__ . '/../../index.php'); wire()->log->save('debug', 'emailpipe.php started'); All I have to do is write the module now!

Hi @BitPoet. include('index.php'); works! EDIT Works from the command line, but not via the pipe🤨

Well, I haven't managed to yet, but I ran the script from the command line and got "Call to undefined function ProcessWire\wire() in /home/xxx/public_html/site/templates/emailpipe.php:4", so it doesn't look like the API is enabled. Script was #!/usr/local/bin/php -q <?php namespace ProcessWire; include_once('../../index.php'); wire()->log->save('debug', 'emailpipe.php started');

Thanks @BitPoet. Doh. You are right, of course, but that crashed the script. I have found what seems to be a little-known fact (certainly not turned up by my researches or by AI): if the piped script (or anything it includes or calls) has CRLF line separators, it will crash. I found a few files in the wire directory that had CRLF (not sure if this was caused by my system or if it is in the PW master). Once I changed these to LF, the script ran with include('../../index.php'). However, attempting to use the API then caused the script to crash - even something really simple like $mailReceivedPage = new Page(); As far as I can see, all the files are now LF. It's really difficult to debug as the only error message I can get is the delivery report, which says: Child process of cagefs_virtual_address_pipe transport returned 255 (could mean shell command ended by signal 127 (Unknown signal 127)) from command: /bin/cagefs_enter There's no easy way of telling exactly what crashed it. Any ideas on debugging this? I am thinking of making a module as an alternative (push vs pull) version of ProcessEmailToPage (if I ever get this working). I will need to remember to warn users that all files must use LF!

I have been using ProcessEmailToPage for quite a while (heavily modified) but realise that it is not really supported plus it is rather OTT for my needs. I was looking at doing a much simpler script. The idea is this - to pipe emails bvia cPanel's 'Default address' facility and then use the script to route them appropriately in PW. This has the huge advantage that it is not necessary to define the email addresses in cPanel and then set up a separate handler for each one in PW. The piping part works fine, and so does the email parsing as far as I can tell. However, I am having difficulty accessing the PW API. I read somewhere (I forget where) that it should be possible to place the script in site/templates/ with the ProcessWire namespace (and maybe include ./index.php ?) but I can't get any of that to work. The 'include' works OK, in that it does not break the script, but adding anything like wire() or 'new Page()' causes the script to fail. Grateful for any suggestions!

For @rastographics and anyone else interested. I believe I have fixed the upgrade issues (or at least I have fixed them satisfactorily for 3 of my sites). Version 2.0.25 is now in the modules library. As well as fixing upgrade() there are also a small number of other minor fixes (including those made in v2.0.24 which did not upgrade properly). Before upgrading, I suggest you copy your existing working module (to, say, .ProcessDbMigrate.versionnumber) in case the fixes do not work in your set-up. Please let me know of any issues (or, indeed, success 😀)

Hi @rastographics. I think the problem is in the upgrade method itself and is unrelated to the changes I made in v2.0.24. It may be that there are spurious files in the system which confuse the upgrade. Can you take a look in modules/ProcessDbMigrate/bootstrap/ and templates/DbMigrate/migrations/bootstrap/ ? They should each have two folders - /new/ and /old/ - each of which has two files - data.json and migration.json and nothing else. Sometimes Windows (😒) adds extra files with the suffix 'Zone.Identifier': I am going to do a new version which ignores those. Can you do a comparison beteween the respective files? i.e. Compare modules/ProcessDbMigrate/bootstrap/new/data.json with templates/DbMigrate/migrations/bootstrap/new/data.json etc. Are there any differences? I have managed to replicate your problem where there are differences between the admin path names - it would be helpful to know if that is the cause of your differences too; I am going to work on a fix for that.

Maybe. I didn't do much but, given the complexity of the module, it is quite easy to accidentally break something when fixing something else. I suggest you stick with 2.0.23 for now while I investigate.

Firstly (after taking a backup), try updating to the latest version 2.0.24 which fixes a few bugs and should be a bit faster. Second, try and ascertain the cause of the problem, by using the preview on the bootstrap page to identify the differences. You might be able to fix them manually in the database. Otherwise, let me know what it says. Lastly, as a temporary fix, try unchecking 'Auto-install bootstrap' on the module config page and seeing whether it works OK.

Sorry - I misunderstood the original comment. However it did cause me to investigate the other issue... That was exactly what I was thinking - and that 'getFresh' might fix it. Based on a bit of testing, it seems like your 2 changes do the job 😀

Just been playing with this. If I edit a field in a child (in a modal) and click 'save' within the modal, then re-edit the child in a modal without saving the parent, then it seems like the change is not saved, regardless of any viewing that might or might not have taken place. In detail: If I edit the child, change a field, save it and close the modal, then save the parent without re-opening the child edit then the change works. If I edit the child, change a field, save it and close the modal, then re-open the child for editing or viewing (in modal or new tab), sometimes* the change has not been made. If I close it again without changing anything, then save the parent then the change is seen not to have worked. *"Sometimes": It seems that if this is the first time I have edited a child then the change does not persist without saving the parent before opening the child again. However, if I have previously edited a child in a table (and saved the parent) then edit it again, it seems like the change does persist after saving the child, without re-saving the parent, regardless of any viewing activity. I'm not sure what is going on here, but it doesn't seem to be anything to do with whether viewing is in a modal or new tab. PS. Sometimes, in situation 2, I get a 'do you want to leave' message when trying to save the parent, but I haven't worked out exactly which combination of circumstances gives rise to that.

I guess that makes sense but is not fully consistent with page list which would require a long click for modal edit too. I'm not sure I understand that - I can't replicate it. In what circumstances is it a problem?

Hi @adrian. Is there any reason why, in the batch child editor, 'edit' opens in a modal but 'view' opens with a new tab. I would prefer consistency (with view opening in a modal) and have done a simple hack to replace the link (target="_blank") with a modal link (class="pw-modal") but obviously that is not a good answer. I appreciate that changing it might upset others (and there may be a good reason why you did that way). Perhaps it could be configurable (panel, modal , new) as in the main page edit?

Done.

Hi @adrian. Thanks for the comments. That was the intention, but I missed it in one important place - hopefully now fixed. I think my new version handles this. Ditto. My code only replaces config items which have page, field, template, role, or user in the key name (case insensitive) and where the value is numeric and is a valid id for the related type. Hopefully this avoids the problem you mention. If someone builds a module with a type name included in the config key with numeric values which are valid ids, but where the values are not intended to refer to ids of that object type, then it won’t work, but that just seems really perverse to me! I have uploaded the new code to my GitHub and have incremented the version and amended the ReadMe. Obviously it is pretty hard to test it on all modules, certainly as regards importing, but it seems ok so far. Let me know if you find any exceptions. One module that causes problems is (deprecated?) ProcessEmailToPage but it is a problem even without my changes. That is because the configdata is wrong - it self-overwrites - so not something that your module can fix, I think.

Hi @adrian (and @Robin S if interested). I have forked your module and added a config field to use names/paths rather than ids. Fork is here: https://github.com/MetaTunes/ModuleSettingsImportExport, if you would like to have a look/play. I have tested it with some modules (but not all) and you can easily see the results in the multiple module settings box. Note that it deliberately gives some warning/error messages at the moment. They are not really necessary but do highlight where some modules are not perfectly constructed. Enjoy 😀