MarkE

Yeah, with PipeEmailToPage you don't need a password, because it is 'push' not 'pull'. You can keep the existing mailboxes or just use the default forwarder.

v1.0.4 is now available in the modules library. This is a major enhancement of the original with SPF/DKIM checking and a lot of functionality built into the admin page (see screenshots below). As mentioned before, there were two motivations for this - partly to move off ProcessEmailToPage which relied on the no-longer-supported flourish library but, more importantly, to allow all email forwarding to be handled in the PW back end rather than having to set up forwarders in cPanel. The users of my 'club membership' app needed to be able to redirect forwarders, but I did not want to give them access to cPanel! This version is now in use in two live sites and seems reasonably robust, so I have upgraded it to 'beta' status, but please do test and monitor. The module is designed to fit in to your own custom web app. It does not do anything interesting unless you have designed the 'parent' and 'mail received' templates and programmed what you want to happen after mails are stored. That said, the admin page does provide a simplified 'inbox'. I will be looking to add a 'reply' capability shortly, but again that will only work if you have an email template defined and a way of sending emails (WireSmtp or similar). The module readme is fairly comprehensive, so refer there for full configuration and usage details. Below are a few illustrative screens. Module config screen (part 1): Defines the 'parents' templates and the email field to match with incoming emails. Plus the email template for pages to store the emails (under the related parent). Config screen part 2: Black/white lists, SPF/DKIM checking, email file retention periods (i.e. retention of the raw incoming email file) and listerpro lister if available. Example admin page. This shows mails which have been successfully processed to pages under parents. The other tabs are for 'unprocessed' emails. ...for example, the quarantine tab shows emails that have failed SPF/DKIM checks, blacklists or the hookable checkSender() method: The maintenance/troubleshooting section separately shows processed emails with no matching page ('orphans') which may occur if the page has been trashed or deleted. It also shows (not in this illustration) if any emails are stuck in the processing queue (this may happen if an error - whether in this module or elsewhere - has caused LazyCron to lock) and provides a button to unblock it. Please ask if you have any questions (but not before reading the readme 😉) and report any bugs.

That’s great. Just what I need to implement a sensible pagination of a member name list. Would that be available for any paginated array?

Hi @Juergen. This is a nice module, but unfortunately it becomes unusable when there are a lot of pages. This is because on every page edit it autoloads and runs getParentPages() which has the following: return wire('pages')->find('id!=' . $exclude_pages . ',templates_id=' . $templates); On one site of mine this takes 20 seconds to run - so that's how long it takes to load a page in the back end. I'm only using the publish/unpublish functionality, so I disabled everything else and it loaded in a flash. However, that's not a solution if you want an 'after' action, so I looked into it a bit more. On my problem site, I have lots of parents and templates but I am only using the module on 2 templates which have a total of 47 pages, so I would have thought it would be possible to write it more efficiently. The problem seems to be that the above code is run on every page edit, not just pages of templates where the 'jk_...' fields are enabled. That means that, unless restricted parents are defined for every parent, the 'find' will run across all pages, with a large template array to match. I suggest adding the following after line 224 if(!in_array($this->page->template->id, array_keys($this->input_templates))) { return new PageArray(); } Then if, for the relevant templates, the allowed parents are set, it all runs pretty swiftly. EDIT: Ideally it might be nice to use this, as then the jk_move_child field could be removed from the template if not required and there would be no need to specify allowable parent templates (which may not be desirable for other reasons) if(!in_array($this->page->template->id, array_keys($this->input_templates)) || !$this->page->hasField('jk_move_child')) { return new PageArray(); } However, re-submitting the module will reinstate the jk_move_child field. A bit more flexibility here would be nice. It would be better if 'move child' was an optional extra.

That works very well. Thanks very much @elabx. It will be incorporated in the next version of PipeEmailToPage.

Simple question: Is there an easy way to have pagination within a MarkupAdminDataTable, or do I have to hand code something?

Module now released - see Interested in any feedback @BitPoet

Season's greetings ProcessWirers. My gift to you is a new module called PipeEmailToPage which you can get from GitHub here: https://github.com/MetaTunes/PipeEmailToPage/tree/main. Fairly 'alpha' at the moment, but I have been using it successfully on a live site for a week or so. I'll test it some more before releasing it to the modules library. I designed this as a replacement for ProcessEmailToPage, which relies on the flourish library which is no longer maintained. It also has a fundamentally different method of operation in that it is 'push' rather than 'pull'. The email addresses are created 'virtually' in the sense that they are not necessarily real email addresses but are held in pages whose templates are linked in the module config. The module works when emails are piped to a script (emailpipe.php) on the server which processes the email and creates the page. The pipe needs to be defined in your hosting service's cPanel or similar. If you define the pipe in the 'Default address' in cPanel, all emails sent to the domain will be processed unless they are specified separately. The 'to' addresses will be matched against the email addresses defined in 'parent' pages holding the virtual addresses. If you define the pipe for a specific email address, only emails sent to that address will be processed (but you can define the same pipe for multiple addresses). A major advantage of the module is that the email addresses can be defined entirely within PW. This means that they can be maintained by someone with no access to cPanel once the developer has set up the pipe. Further details are in the readme. I suggest you test thoroughly before using (and take careful note of the 'points to note' in the readme). Turn on some of the logging statements if you wish. Also, check your mail delivery in cPanel to look for errors there. If you report errors to me, I will do my best to hunt them, but please include as much info as possible. I will also be grateful for any suggested code improvements and/or PRs. 🎄

Apologies to anyone ( @BitPoet, @teppo ?) who may have downloaded the module - I left some code in by mistake which crashed it. Now hopefully fixed.

Also, it would be nice to fix the softaculous text to not say "ProcessWire is a free PHP5 content management system and framework ". That is a bit of a turn-off.

Hi @BitPoet and anyone else interested. An initial release of this module is available here: https://github.com/MetaTunes/PipeEmailToPage Still quite a lot of refinements to add (and no doubt some bugs to fix), including adding a queue as discussed above. Also you need to install the zbateson parser with composer at the moment. Any feedback is appreciated.

My planned approach to this was to allow multiple templates as 'parents' for child mails. The mail will be saved under whichever parent has the matching 'to' address. It is then up to the user to use a LazyCron to process the mail pages as they wish (with different processing possible for different parent templates). So, for example, 'blog@mydomain.com' can be used to process to a blog whereas 'maillist@mydomain.com' can forward the mail to a list.

That's what I feared and that suppression might not fully work. Good idea. I'll look into that. Definitely. I am trying to build it to be as general-purpose as possible, although my immediate need is to replace a hacked version of ProcessEmailToPage in a specific application. Hence the provision of hooks. Once I have the components in place, I will put it on GitHub. My view is that open source benefits everyone, including the original developer.

Thanks @BitPoet. That’s really helpful. I’ve pretty much decided on Mime-Mail-Parser for the reasons you state. So far I’ve installed it with composer, but will reconsider that. All the basics are now working nicely 😀. This includes a hookable method to allow people to add custom processing to check sender credentials. I will add whitelists and blacklists and am looking at additional anti-spam techniques. I also have quite a bit of work to do on the module config and Process page. Re queuing, I had considered that, but ditched it in favour of more rapid immediate processing. However, the point you raise about database availability is a good one as it might cause emails to just get lost. Maybe overcomplicating things, but how about a try-catch, just queuing it if there is a failure? This would need to be carefully crafted, as any error messages would result in the pipe failing. The advantage of always queuing is that it avoids any risk of errors being generated from attempting to access PW. I guess a 1 minute delay is acceptable but the instantaneous processing has been helpful while debugging.

I am looking at making a replacement/alternative module for ProcessEmailToPage for 2 reasons: I want something where a webmaster can set up the process without needing access to cPanel to set up new email addresses. The ProcessEmailToPage module is not really supported any more because it relies on Flourish, which is 12 years old and is no longer maintained. I'm sure @Petedoesn't mind! The main difference is that PipeEmailToPage will be a 'push' rather than a 'pull' approach. In cPanel, you will set up a pipe to a php file for 'Default addresses' (so they do not have to be specifically defined). The php file will be something like /site/modules/PipeEmailToPage/emailpipe.php. Thanks to some help from @BitPoet, I have the pipe working. There are a few gotchas to watch for, like files with CRLF not LF, and you need to set the emailpipe.php to execute permission 755, which it would not normally have. Then there will be a module script to handle various settings and maybe a Process page to view emails. My next step is to add an email parser. There are a number out there. In particular https://github.com/zbateson/mail-mime-parser and https://github.com/php-mime-mail-parser/php-mime-mail-parser, but maybe others too. Each of those installs via composer; however, I'm not sure I want that, as all the code for the module should be packaged within it, I think, rather than rely on external packages. Also the second of those two requires php extensions that may not be installed (but are probably not difficult to add). I would appreciate any thoughts on (a) php email parsers and (b) the use of composer vs including the code in the module. Or anything else relavant to this project 🙂.

This works: #!/usr/local/bin/php -q <?php namespace ProcessWire; include(__DIR__ . '/../../index.php'); wire()->log->save('debug', 'emailpipe.php started'); All I have to do is write the module now!

Hi @BitPoet. include('index.php'); works! EDIT Works from the command line, but not via the pipe🤨

Well, I haven't managed to yet, but I ran the script from the command line and got "Call to undefined function ProcessWire\wire() in /home/xxx/public_html/site/templates/emailpipe.php:4", so it doesn't look like the API is enabled. Script was #!/usr/local/bin/php -q <?php namespace ProcessWire; include_once('../../index.php'); wire()->log->save('debug', 'emailpipe.php started');

Thanks @BitPoet. Doh. You are right, of course, but that crashed the script. I have found what seems to be a little-known fact (certainly not turned up by my researches or by AI): if the piped script (or anything it includes or calls) has CRLF line separators, it will crash. I found a few files in the wire directory that had CRLF (not sure if this was caused by my system or if it is in the PW master). Once I changed these to LF, the script ran with include('../../index.php'). However, attempting to use the API then caused the script to crash - even something really simple like $mailReceivedPage = new Page(); As far as I can see, all the files are now LF. It's really difficult to debug as the only error message I can get is the delivery report, which says: Child process of cagefs_virtual_address_pipe transport returned 255 (could mean shell command ended by signal 127 (Unknown signal 127)) from command: /bin/cagefs_enter There's no easy way of telling exactly what crashed it. Any ideas on debugging this? I am thinking of making a module as an alternative (push vs pull) version of ProcessEmailToPage (if I ever get this working). I will need to remember to warn users that all files must use LF!

I have been using ProcessEmailToPage for quite a while (heavily modified) but realise that it is not really supported plus it is rather OTT for my needs. I was looking at doing a much simpler script. The idea is this - to pipe emails bvia cPanel's 'Default address' facility and then use the script to route them appropriately in PW. This has the huge advantage that it is not necessary to define the email addresses in cPanel and then set up a separate handler for each one in PW. The piping part works fine, and so does the email parsing as far as I can tell. However, I am having difficulty accessing the PW API. I read somewhere (I forget where) that it should be possible to place the script in site/templates/ with the ProcessWire namespace (and maybe include ./index.php ?) but I can't get any of that to work. The 'include' works OK, in that it does not break the script, but adding anything like wire() or 'new Page()' causes the script to fail. Grateful for any suggestions!

For @rastographics and anyone else interested. I believe I have fixed the upgrade issues (or at least I have fixed them satisfactorily for 3 of my sites). Version 2.0.25 is now in the modules library. As well as fixing upgrade() there are also a small number of other minor fixes (including those made in v2.0.24 which did not upgrade properly). Before upgrading, I suggest you copy your existing working module (to, say, .ProcessDbMigrate.versionnumber) in case the fixes do not work in your set-up. Please let me know of any issues (or, indeed, success 😀)

Hi @rastographics. I think the problem is in the upgrade method itself and is unrelated to the changes I made in v2.0.24. It may be that there are spurious files in the system which confuse the upgrade. Can you take a look in modules/ProcessDbMigrate/bootstrap/ and templates/DbMigrate/migrations/bootstrap/ ? They should each have two folders - /new/ and /old/ - each of which has two files - data.json and migration.json and nothing else. Sometimes Windows (😒) adds extra files with the suffix 'Zone.Identifier': I am going to do a new version which ignores those. Can you do a comparison beteween the respective files? i.e. Compare modules/ProcessDbMigrate/bootstrap/new/data.json with templates/DbMigrate/migrations/bootstrap/new/data.json etc. Are there any differences? I have managed to replicate your problem where there are differences between the admin path names - it would be helpful to know if that is the cause of your differences too; I am going to work on a fix for that.

Maybe. I didn't do much but, given the complexity of the module, it is quite easy to accidentally break something when fixing something else. I suggest you stick with 2.0.23 for now while I investigate.

Firstly (after taking a backup), try updating to the latest version 2.0.24 which fixes a few bugs and should be a bit faster. Second, try and ascertain the cause of the problem, by using the preview on the bootstrap page to identify the differences. You might be able to fix them manually in the database. Otherwise, let me know what it says. Lastly, as a temporary fix, try unchecking 'Auto-install bootstrap' on the module config page and seeing whether it works OK.

Sorry - I misunderstood the original comment. However it did cause me to investigate the other issue... That was exactly what I was thinking - and that 'getFresh' might fix it. Based on a bit of testing, it seems like your 2 changes do the job 😀