Module ProcessEmailToPage

[huseyin]

I recieve SPAM emails with email addresses like **name**@domain.com.

For such emails the the module throws error messeges and stops reading further. No emails after this SPAM are downloaded.

The error shown in Tracy Debugger is like this : 

Unknown Selector operator: '=**' -- was your selector value properly escaped? field='email', value='DialVision**@carsvasite.us', selector: 'email=**DialVision**@carsvasite.us, sort=sort, parent_id=29, templates_id=3' (in /wire/core/Selectors.php line 419)

How can i bypass this error...

[MarkE]

On 1/28/2020 at 3:04 PM, huseyin said:

Unknown Selector operator: '=**' -- was your selector value properly escaped?

This doesn't look like a module issue. Try using single quotes around the double asterisks: '**' not **

i.e. (in your example) email='**DialVision**@carsvasite.us'

EDIT: Correction - in my off-the-cuff response I hadn't realised that this is in the code, not set in options. See the link in the next post.

[MarkE]

It's a shame that this module is no longer supported. I have an app that uses it and it works well. I'm not sure that the use of Flourish is a problem as it only uses a few classes, which presumably could be modified if they cause issues  in the future with new PHP versions. I can't find a better alternative at the moment.

However, there are a couple of problems I have had with the module:

• html tags such as <ol> cause problems
• The additional "notify users" for categories does not work well. Firstly, any additional users in one category are applied to other categories and, secondly, only one set of From, Subject and Body is allowed.

I have attempted to fix these at https://github.com/MetaTunes/ProcessEmailToPage. Please feel free to use (and to fix any further bugs!). I'm not promising super support, but will look into any genuine bugs.

Another point to note re multiple categories - all the email addresses must use the same password as this is set at an overall level. I have not attempted to fix this.

EDIT: updated code to fix the issue reported by @huseyin - let me know if it works.

[adrian]

@MarkE - thanks for taking up the torch and trying to get some things fixed. Perhaps you could could get in touch with @Pete and get access to push changes to the main repo, or perhaps have the modules directory point to your forked version?

Again, sorry I can't keep working on it, but I've never actually used it, so that makes it hard to justify the time, not to mention that I really don't know what is and isn't needed because I don't see the problems as they arise.

[Pete]

@MarkE I may finally be using this module on a project many years after last being involved with it. Did you want access to the main repo to merge your changes into it?

My needs may be somewhat different - updating a trip diary (single page, many entries) over the course of a trip - so I'd likely be using a modified version anyway but whilst I'm looking back at this topic I thought I'd ask ? 

[Pete]

I think I've actually already spotted some workflow flaws that mean I couldn't use it for this project actually, but the offer of access/managing the repo still stands.

[adrian]

@Pete - in case it helps, I had to do something with processing incoming emails recently and given the abandones status of the flourish library, I went with pure PHP. 

$inbox = imap_open('{imap.gmail.com:993/imap/ssl/novalidate-cert}', 'me@gmail.com', 'password');
$msgnos = imap_search($inbox, 'UNSEEN');

if($msgnos) {
    foreach($msgnos as $msgno) {
        
        imap_setflag_full($inbox, $msgno, "\\Seen \\Flagged", ST_UID);
        
        $header = imap_headerinfo($inbox, $msgno);
        
        $bodyText = imap_fetchbody($inbox, $msgno, 1.2);
        if(!strlen($bodyText) > 0) {
            $bodyText = imap_fetchbody($inbox, $msgno, 1);
        }
        $timestamp = strtotime($header->date);
        $toObj = $header->to[0];
        $fromObj = $header->from[0];
        $to = $toObj->mailbox.'@'.$toObj->host;
        $from = $fromObj->mailbox.'@'.$fromObj->host;
        $u = $users->get('email='.$from);
        if($u->id) {
            // process contents $bodyText here
        }
    
    }

}

I had this all ready to use and then decided to actually switch to using Mailgun's incoming callback feature - if you have access to it, I think it's much nicer because they will ping your site via a webhook callback, rather than you needing to regularly ping your email via imap.

[MarkE]

@Pete - the module has been working trouble-free for the last 2 years. I’ve pretty much forgotten the details, but will look into it when I have a mo. Funnily enough, I may have another app when I might want to use it - but I’ll look at @adrian’s solution as well. Mailgun would be good for the first app also, but not the second, so that’s a useful lead too, thanks.  

[tires]

By visiting the hash URL to checking the mail manually i get the following error message:

Deprecated: strpos(): Non-string needles will be interpreted as strings in the future. Use an explicit chr() call to preserve the current behavior in /mydomain/mysite/site/modules/ProcessEmailToPage/flourishlib/fMailbox.php on line 505

How can i avoid this?

[adrian]

@tires - the Flourish library is no longer maintained, so this module really needs to be rebuilt so that it doesn't rely on it. It's quite easy with imap_open https://www.php.net/manual/en/function.imap-open.php but I'm afraid I really can't contribute more time to this module - I've never actually used it myself and I don't think it has a huge user base, but I am sure Pete would accept a PR if you'd be willing to implement imap_open.

 

 

[MarkE]

I use it on one site and it is quite neat, but I have yet to move that site to php8, so thanks for the heads-up @tires

[tires]

57 minutes ago, adrian said:

@tires - the Flourish library is no longer maintained, so this module really needs to be rebuilt so that it doesn't rely on it. It's quite easy with imap_open https://www.php.net/manual/en/function.imap-open.php but I'm afraid I really can't contribute more time to this module - I've never actually used it myself and I don't think it has a huge user base, but I am sure Pete would accept a PR if you'd be willing to implement imap_open.

It would be a pity if the module was not continued!
I'm afraid I can't contribute in any helpful way. Unfortunately i lack the skills ...

[adrian]

In case someone feels like contributing by replacing Flourish with imap_open, this code should get you started:

$inbox = imap_open('{imap.gmail.com:993/imap/ssl/novalidate-cert}', 'address@domain.com', 'password');
$msgnos = imap_search($inbox, 'UNSEEN');

if($msgnos) {
    
   foreach($msgnos as $msgno) {

       imap_setflag_full($inbox, $msgno, "\\Seen \\Flagged", ST_UID);

       $header = imap_headerinfo($inbox, $msgno);

       $bodyText = imap_fetchbody($inbox, $msgno, 1.2);
       if(!strlen($bodyText) > 0) {
           $bodyText = imap_fetchbody($inbox, $msgno, 1);
       }
       $timestamp = strtotime($header->date);
       $toObj = $header->to[0];
       $fromObj = $header->from[0];
       $to = $toObj->mailbox.'@'.$toObj->host;
       $from = $fromObj->mailbox.'@'.$fromObj->host;

   }

}

 

[MarkE]

On 10/18/2022 at 9:58 PM, MarkE said:

I use it on one site and it is quite neat, but I have yet to move that site to php8

I've now moved that site to php 8 and the module has stopped working. Any insights would be great. Meanwhile I will try and diagnose.

[MarkE]

1 hour ago, MarkE said:

Meanwhile I will try and diagnose.

The problem (or, at least, on of them) is line 505 of fMailbox.php. Since php 8.0.0, providing a non-string as a needle to strpos is no longer supported. $filter has a default value of null. 

(strpos($header, $filter) !== FALSE) evaluates to true and should be replaced. The following works in my case, but further fixes may be required if the $filter is actually set.

if ($filter && strpos($header, $filter) !== FALSE)

 

[adrian]

@MarkE - I really do think migration to use imap_open as per my above post would be a better option given the status of the flourish library.

[MarkE]

Just now, adrian said:

I really do think migration to use imap_open as per my above post would be a better option given the status of the flourish library.

You are almost certainly right. I needed a quick fix! I'm using POP3 at the moment. I'll try your suggestion when I have a bit more time. Ta.

[MarkE]

Hi @adrian, @Pete & @tires and anyone else interested in this module. As I said above, I was using this on one site and am now looking at using it more widely as it is very useful for sending emails to a group: the email is saved to a page then forwarded (via LazyCron) to a list, a record thus being maintained of all such emails.

Generally it has worked well without the suggested imap upgrade. However, I recently encountered a problem. The email import is triggered by a hook on Page render. My new pagebuilder module (not released) uses multiple PW pages per displayed page. I think the rapid-fire execution caused the emails to be picked up multiple times without the server deleting them. I tried replacing the hook, without success, with a LazyCron hook. Instead I have just added a session var so that the import cannot happen more than once a minute.

I'm afraid my technical understanding of the cause of this problem is (very) limited, but I wonder whether it would also be an issue with imap...

[MarkE]

Another issue for anyone else using this module: Line 625 of ProcessEmailToPage.module has 

$attachment['filename'] = $this->sanitizer->pageName($attachment['filename'], true);

I don't know why the pageName sanitizer is being used to sanitize a file name, but this gives an incorrect result if the file name has brackets in it. In my version, I have changed pageName to fileName.

[tires]

With my new mail server, this module no longer works for me either. That's a real shame, because it's a great and helpful module.
I assume that you only have to replace the flourish components with the imap_open function. Most of the code should remain untouched.

I also lack the technical expertise.
The AI suggests the following code:

<?php

/**
 * ProcessEmailToPage.module
 * Module to fetch emails from a mailbox and convert them into ProcessWire pages
 */

class ProcessEmailToPage extends WireData implements Module {
    public $moduleHash;
    public $emailHost;
    public $emailPort;
    public $emailPassword;
    public $emailType;
    public $forceSecure = false;
    public $categoryData;

    // Module info
    public static function getModuleInfo() {
        return array(
            'title' => 'Process Email to Page',
            'version' => 1,
            'summary' => 'Fetches emails from a mailbox and creates pages from the messages.',
            'author' => 'Your Name',
            'requires' => 'ProcessWire 3.x'
        );
    }

    public function init() {
        // Set the hash for security to ensure only authorized requests are processed
        $this->moduleHash = wire('config')->hash;

        // Example category data (JSON)
        $this->categoryData = '{"categories": [{"emailAddress": "your-email@example.com"}]}'; 

        // Set connection parameters
        $this->emailHost = 'imap.example.com'; // Example IMAP server
        $this->emailPort = 993; // SSL port
        $this->emailPassword = 'your-password'; // Your email password
        $this->emailType = 'imap'; // Protocol type (imap/pop3)
    }

    public function execute() {
        // Process emails if the correct hash is provided
        if(wire('input')->get('hash') && wire('input')->get('hash') == $this->moduleHash) {
            // Decode category data from JSON into an object
            $categories = json_decode($this->categoryData);

            // Iterate through each email category
            foreach($categories as $category) {

                // Set up email connection
                $this->emailType = empty($this->emailType) ? 'imap' : strtolower($this->emailType); // Default to 'imap'
                $server = '{' . $this->emailHost . ':' . ($this->emailPort ? $this->emailPort : 993) . '/imap/ssl}INBOX'; // IMAP server with SSL
                $username = $category->emailAddress;
                $password = $this->emailPassword;

                // Connect to the mail server using imap_open
                $mailbox = imap_open($server, $username, $password);

                if (!$mailbox) {
                    echo "Error connecting to the mail server: " . imap_last_error();
                    continue;
                }

                // Search for all emails (can be adjusted to 'UNSEEN' for unread emails)
                $emails = imap_search($mailbox, 'ALL');

                if ($emails) {
                    rsort($emails); // Sort emails with the newest first
                    $messages = [];

                    // Iterate through each email ID
                    foreach($emails as $email_id) {
                        // Fetch the email header information
                        $overview = imap_fetch_overview($mailbox, $email_id, 0);
                        $messages[] = $overview[0]; // Store header information

                        // Optional: Output more header fields such as "From", "Subject", "Date"
                        // echo "Subject: " . $overview[0]->subject . "\n";
                        // echo "From: " . $overview[0]->from . "\n";
                        // echo "Date: " . $overview[0]->date . "\n";
                    }

                    // Now process the emails, for example, create a page for each email
                    foreach ($messages as $message) {
                        $page = $pages->add('email');
                        $page->title = $message->subject;
                        $page->body = "Email from: " . $message->from . " on " . $message->date;
                        $page->save();
                    }

                    // Delete processed emails from the server
                    foreach ($emails as $email_id) {
                        imap_delete($mailbox, $email_id);
                    }

                    // Expunge to permanently remove the deleted emails
                    imap_expunge($mailbox);
                }

                // Close the connection to the mail server after processing
                imap_close($mailbox);
            }
        }
    }
}

 

[tires]

I have just realized that the module is still running with my new mailboxes. The problem was that I have several mailboxes that I retrieve. And since there is only one field for the password, this must be the same for all mailboxes ...

[tires]

It seems to continue there!
Great!!!

[MarkE]

On 1/21/2025 at 11:31 AM, tires said:

And since there is only one field for the password, this must be the same for all mailboxes ...

Yeah, with PipeEmailToPage you don't need a password, because it is 'push' not 'pull'. You can keep the existing mailboxes or just use the default forwarder.