BitPoet

Nice. I didn't think of having "intermediate" mail pages for further processing, but it does make sense.

The problem is that PHP has a bunch of errors that can't be caught (or better worded, not worked around with custom error handlers - try-catch only catches Exceptions, which are just one side of the medal). The number of unrecoverable errors has been dwindling while version numbers rose, but there are still a few left, and they tend to show up unexpected. Re queueing or not: you can always implement both. It could even come with its own script. If the filtering and page creation is handled by methods in your module, it should even be a a piece of cake. Are you planning to open source your module? If yes, I could implement the feature and send a pull. The thought behind that is that there are different ways to deliver mails and different reasons to add them as pages, from posting blog content to implementing a support ticket system on top of PW (an idea that has been nagging me for a long time), and probably a lot of others.

I get the same error, but I can install just fine. I don't think the dev environment plays a role there, it's the browser running the JS. Why is the error preventing you from continuing? The installation page should still be there. However, you should be able to comment out line 843 in wire/modules/AdminTheme/AdminThemeUikit/scripts/main.js with no ill effects if Safari for some weird reasons prevents the page from working because of the error.

It's always a tricky decision whether to include packages or have the user run composer. If license terms allow it, I've so far included the full source to make sure users don't end up with incompatibilities introduced by future versions. This, of course, sometimes means to react quickly if a vulnerability is discovered in a package my module relies upon, but I still prefer that to dysfunctional installations. About using parser libraries at all - PHPs native capabilities are a PITA, and there are reasons why almost everybody uses a third party parser package (I had to rely on native stuff for a bunch of email parsing scripts that extracted contents of non standard conformant emails, and that was no fun at all). Which to choose? A hard decision. Both are mature projects. One thing that comes to mind would be the dependency on PECL mailparse, which also has a known memory leak that can hit hard if one tries to process a larger number of mails in a loop. This and more intuitive handling of addresses could be a point towards Mail-Mime-Parser. On the other hand, a few content part handling methods feel more intuitive for PHP-Mime-Mail-Parser. No clear winner there, but I'd personally go for the simpler option from the user's perspective (i.e. Mail-Mime-Parser). You might consider working with a queue directory as an option instead of using a direct pipe, which would improve resilience and performance. In cases like the database going away, all emails could still be written to the directory and processed later in a (lazy) cron job. Interfacing software is my main job description, and one philosophy I've acquired is to always finish pipe jobs as fast as possible.

Glad you got it working. So it is an include_path issue after all. Looks like the pipe call doesn't change the current working directory, so PHP looks for index.php somewhere in the cPanel file tree.

That's strange. Is "." in include_path (otherwise PHP might not find index.php with a relative path)?

You can enable error logging for php-cli by setting error_log in /etc/php/cli/php.ini to something like "/var/log/php_errors.log". If you have dos2unix on your system, you can recursively convert all files to proper LF line endings from within your PW directory. find . -type f -name '*.php' -exec dos2unix {} find . -type f -name '*.module' -exec dos2unix {}

Inside site/templates, it would have to be include('../../index.php'), though you should be able to place the script anywhere outside the PW directory as long as you specify the correct path to the index.php that lives in PW's root directory. You'll have to make sure, of course, that the user under which cPanel runs has read access to the PW directories. Since php-cgi und php-cli sometimes use different php.ini files (depending on your OS), this may also be worth checking. Otherwise, if you're including the correct index.php and have the namespace set to ProcessWire, wire() and 'new Page()' will work.

Big thumbs up for that idea. If the timing is right, I might even make it to both, since I'm planning to head across the pond and hike the John Muir Trail next summer. Love the preload method. I've got a shop solution built on PW with about 120k orders and 150k order positions by now, all as pages. Performance is becoming a bit of an issue in the backend with long range price change statistics and detailed Excel exports, and this seems like a perfect use case.

WireDatabasePDO's constructur doesn't connect to the database yet. This is either implicitly done when the query or exec(ute) methods are called, or explicitly by invoking $db->pdo(), which will return the underlying plain PDO object.

Yes, I did. Sorry for the mixup. If upping wait_timeout, deleting individual entries from your image field's table is normally an option, but I'm not familiar with ImageExtra. So I'd make sure to have a working database backup before attempting it. One thing you could try is setting the field you want to exclude from saving to hidden.

Is that on a shared hosting server? The error indicates that the execution time of a mysql query was exceeded or a memory limitation in the database was reached, in which case the server aborts the query and drops the connection. Those are often pretty close knit on shared hosts. If you have access to the database itself, the first thing to look at would be wait_timeout. "SELECT @@wait_timeout" will give you the seconds MySQL will spend on a query. If that is very low, you can try setting a higher per-session value through $config->dbInitCommand in site/ready.php. $config->dbInitCommand = "SET NAMES '{charset}'; SET wait_timeout = 300";

The question prompted me to put FieldtypeSketchyDate on GitHub. It's a small module I started writing for a solution that never went anywhere, so its pretty basic. It may be a starting point though. In short, it lets you enter 2024 or 2024/10 or 2024/10/11 in the date value, and it also lets you search for 2024/10 and returns all pages with a date between 2024/10/01 and 2024/10/31 as well as those where 2024/10 was entered. So far, it just validates the formatting through the HTML pattern attribute. Proper date validation and calendar UI are still on the todo list.

As an alternative, but which also involves some coding: you can also create fields + fieldgroup + template + a "storage" page in your module's install method. Then, in your processForm method, you can fetch that page and assign the form values there, or you just display a modal link to the page editor in your buildForm method and let PW take care of saving the values. That approach can get a bit involved (double so if you want to clean up in your uninstall method), but it can have some advantages (retaining the values after module uninstall, straight forward use of arbitrary field types).

The first thing I'd look for would be whether there's an auto_prepend_file entry in php.ini.

The interesting information is probably in the response body for the call to edit/?id=????%InputfieldFileAjax=1.

I bet you didn't forget to assign a sensible page name, then ?

It should be faster with transaction, and I did add startTransaction and commit in my import script. I'm still trying to figure out where the bottleneck is. Shouldn't be the hardware (i9-14900K, 64GB, high speed NVMe SSD), and I didn't see any io waits in the stats, so something is probably wrong with my db server (MySQL 8.4.0 on Windows 11). Looks like I've got do some forensics there.

Yep. About 24 hours on InnoDB for a full import. More that 52k books, 983 genres, 48k characters, 10k publishers, 15k awards. All in all 95314 pages in an otherwise pristine PW installation. I'm attaching a site profile with the data if someone wants to play around. site-bookinventorydata.zip

Most likely an issue with the webserver. Apache's mod_security blocks encoded urls in GET parameters by default, so I'd start looking there.

Just a thought for future in case performance really becomes an issue for someone: it's possible to add a generated + stored column to a MySQL table that feeds itself from a dedicated value in a JSON column, and this column can of course have a fulltext index. A hook on PageFinder::getQueryUnknownField could then "redirect" the query to the generated column. Would only work on single valued fields, of course. For anybody interested in creating such a generated field from a JSON field, here's a blog entry that explains it better than I could.

You should be able to append the required settings to $config->dbInitCommand.

Did you set $config->sessionName to different values?

I'd probably approach that with a hook on User::unpublished, perform the checks there whether the user is staff and referenced from a client account, and send a message (through PW's message system or even other means like a ticket system or email).

In AdminDevModeColors, I hook onto AdminTheme::getExtraMarkup, but that's probably not much different from your approach.