BitPoet

As @dynweb wrote, points one and two can be covered by the pagefileSecure option that is part of the core. Enable $config->pagefileSecure in site/config.php and uncomment the relevant entry in .htaccess so all file requests are routed through index.php.

Point three is difficult, and off the top of my head, I'm not sure if a solution exists. You'd need two different servers to physically separate the encrypted files and the corresponding decryption keys, otherwise, anybody in control of the server only needs a bit of time to locate the keys. In a perfect world, the decryption would happen on the client, so no server ever sees the unencrypted file - otherwise, it would just be a question of patience for the hacker to collect them if they hacked the file delivery server. It should be possible to build client side decryption around fetch API, transformers, webcrypto and perhaps FileSystemAccess (with polyfill for a few browsers), but together with taking care of proper key handling, PW integration of uploads and authentication and authorization across both systems, this would certainly be a complex effort.

I'm immediately thinking of SAML2 auth with group/role mapping to ensure proper authorization. Handling uploads should be possible by hooking into InputfieldFile. PW would need to create a key, hand that over to the key storage server together with a hash of the file url as key and store the encrypted file. InputfieldFile could be extended with a hook method to return a proper URL to retrieve the decryption key. Client side JS would retrieve the file and key URLS, download both and decrypt the file. transcend-io's Penumbra JS library for stream decryption looks promising at first glance.

MAMP Pro should allow you to enable screen output of errors for PHP when you're in expert view. That should give you a better idea of what's going wrong.

There's no direct selector option, but you could get all templates that use the field first, then use those in the selector. Unless you use the field in repeaters or related types, fieldgroup name = template name.

$tpls = $fields->get('checkbox_export')->getFieldgroups()->implode('|', 'name');
$ps = pages()->find("template=$tpls, checkbox_export!=1, sort=id, include=hidden, status<" . Page::statusTrash);

24 minutes ago, MarkE said:

My planned approach to this was to allow multiple templates as 'parents' for child mails. The mail will be saved under whichever parent has the matching 'to' address. It is then up to the user to use a LazyCron to process the mail pages as they wish (with different processing possible for different parent templates). So, for example, 'blog@mydomain.com' can be used to process to a blog whereas 'maillist@mydomain.com' can forward the mail to a list.

Nice. I didn't think of having "intermediate" mail pages for further processing, but it does make sense.

7 hours ago, MarkE said:

how about a try-catch, just queuing it if there is a failure

The problem is that PHP has a bunch of errors that can't be caught (or better worded, not worked around with custom error handlers - try-catch only catches Exceptions, which are just one side of the medal). The number of unrecoverable errors has been dwindling while version numbers rose, but there are still a few left, and they tend to show up unexpected.

Re queueing or not: you can always implement both. It could even come with its own script. If the filtering and page creation is handled by methods in your module, it should even be a a piece of cake. Are you planning to open source your module? If yes, I could implement the feature and send a pull. The thought behind that is that there are different ways to deliver mails and different reasons to add them as pages, from posting blog content to implementing a support ticket system on top of PW (an idea that has been nagging me for a long time), and probably a lot of others.

I get the same error, but I can install just fine. I don't think the dev environment plays a role there, it's the browser running the JS. Why is the error preventing you from continuing? The installation page should still be there. However, you should be able to comment out line 843 in wire/modules/AdminTheme/AdminThemeUikit/scripts/main.js with no ill effects if Safari for some weird reasons prevents the page from working because of the error.

It's always a tricky decision whether to include packages or have the user run composer. If license terms allow it, I've so far included the full source to make sure users don't end up with incompatibilities introduced by future versions. This, of course, sometimes means to react quickly if a vulnerability is discovered in a package my module relies upon, but I still prefer that to dysfunctional installations.

About using parser libraries at all - PHPs native capabilities are a PITA, and there are reasons why almost everybody uses a third party parser package (I had to rely on native stuff for a bunch of email parsing scripts that extracted contents of non standard conformant emails, and that was no fun at all).

Which to choose? A hard decision. Both are mature projects. One thing that comes to mind would be the dependency on PECL mailparse, which also has a known memory leak that can hit hard if one tries to process a larger number of mails in a loop. This and more intuitive handling of addresses could be a point towards Mail-Mime-Parser. On the other hand, a few content part handling methods feel more intuitive for PHP-Mime-Mail-Parser. No clear winner there, but I'd personally go for the simpler option from the user's perspective (i.e. Mail-Mime-Parser).

You might consider working with a queue directory as an option instead of using a direct pipe, which would improve resilience and performance. In cases like the database going away, all emails could still be written to the directory and processed later in a (lazy) cron job. Interfacing software is my main job description, and one philosophy I've acquired is to always finish pipe jobs as fast as possible.

Glad you got it working.

So it is an include_path issue after all. Looks like the pipe call doesn't change the current working directory, so PHP looks for index.php somewhere in the cPanel file tree.

That's strange. Is "." in include_path (otherwise PHP might not find index.php with a relative path)?

You can enable error logging for php-cli by setting error_log in /etc/php/cli/php.ini to something like "/var/log/php_errors.log".

If you have dos2unix on your system, you can recursively convert all files to proper LF line endings from within your PW directory.

find . -type f -name '*.php' -exec dos2unix {}
find . -type f -name '*.module' -exec dos2unix {}

Inside site/templates, it would have to be include('../../index.php'), though you should be able to place the script anywhere outside the PW directory as long as you specify the correct path to the index.php that lives in PW's root directory. You'll have to make sure, of course, that the user under which cPanel runs has read access to the PW directories. Since php-cgi und php-cli sometimes use different php.ini files (depending on your OS), this may also be worth checking. Otherwise, if you're including the correct index.php and have the namespace set to ProcessWire, wire() and 'new Page()' will work.

13 hours ago, wbmnfktr said:

Maybe we should create some kind of meetups in the US and maybe even in the EU next year.
That would be fun.

Big thumbs up for that idea. If the timing is right, I might even make it to both, since I'm planning to head across the pond and hike the John Muir Trail next summer.

Love the preload method. I've got a shop solution built on PW with about 120k orders and 150k order positions by now, all as pages. Performance is becoming a bit of an issue in the backend with long range price change statistics and detailed Excel exports, and this seems like a perfect use case.

WireDatabasePDO's constructur doesn't connect to the database yet. This is either implicitly done when the query or exec(ute) methods are called, or explicitly by invoking $db->pdo(), which will return the underlying plain PDO object.

6 hours ago, MichaPau said:

did you mean site/config.php

Yes, I did. Sorry for the mixup.

If upping wait_timeout, deleting individual entries from your image field's table is normally an option, but I'm not familiar with ImageExtra. So I'd make sure to have a working database backup before attempting it.

One thing you could try is setting the field you want to exclude from saving to hidden.

Is that on a shared hosting server? The error indicates that the execution time of a mysql query was exceeded or a memory limitation in the database was reached, in which case the server aborts the query and drops the connection. Those are often pretty close knit on shared hosts. If you have access to the database itself, the first thing to look at would be wait_timeout. "SELECT @@wait_timeout" will give you the seconds MySQL will spend on a query. If that is very low, you can try setting a higher per-session value through $config->dbInitCommand in site/ready.php.

$config->dbInitCommand = "SET NAMES '{charset}'; SET wait_timeout = 300";

The question prompted me to put FieldtypeSketchyDate on GitHub. It's a small module I started writing for a solution that never went anywhere, so its pretty basic. It may be a starting point though.

In short, it lets you enter 2024 or 2024/10 or 2024/10/11 in the date value, and it also lets you search for 2024/10 and returns all pages with a date between 2024/10/01 and 2024/10/31 as well as those where 2024/10 was entered. So far, it just validates the formatting through the HTML pattern attribute. Proper date validation and calendar UI are still on the todo list.

22 hours ago, poljpocket said:

You must create your own database schema and save your data there if that is what you are after.

As an alternative, but which also involves some coding: you can also create fields + fieldgroup + template + a "storage" page in your module's install method. Then, in your processForm method, you can fetch that page and assign the form values there, or you just display a modal link to the page editor in your buildForm method and let PW take care of saving the values. That approach can get a bit involved (double so if you want to clean up in your uninstall method), but it can have some advantages (retaining the values after module uninstall, straight forward use of arbitrary field types).

The first thing I'd look for would be whether there's an auto_prepend_file entry in php.ini.

The interesting information is probably in the response body for the call to edit/?id=????%InputfieldFileAjax=1.

1 hour ago, da² said:

In my current project I sometimes import 5000 pages using PW API, and with transactions (wire()->database->beginTransaction(); wire()->database->commit();) it is done in 20-30 seconds.

I bet you didn't forget to assign a sensible page name, then ?

1 hour ago, da² said:

In my current project I sometimes import 5000 pages using PW API, and with transactions (wire()->database->beginTransaction(); wire()->database->commit();) it is done in 20-30 seconds.

It should be faster with transaction, and I did add startTransaction and commit in my import script. I'm still trying to figure out where the bottleneck is. Shouldn't be the hardware (i9-14900K, 64GB, high speed NVMe SSD), and I didn't see any io waits in the stats, so something is probably wrong with my db server (MySQL 8.4.0 on Windows 11). Looks like I've got do some forensics there.

8 hours ago, wbmnfktr said:

The most fun part until now was to realise that importing 50k items is something different than handling 50k items that grew over years. A nice reality check.

Yep. About 24 hours on InnoDB for a full import. More that 52k books, 983 genres, 48k characters, 10k publishers, 15k awards. All in all 95314 pages in an otherwise pristine PW installation.

I'm attaching a site profile with the data if someone wants to play around.

site-bookinventorydata.zip

Most likely an issue with the webserver. Apache's mod_security blocks encoded urls in GET parameters by default, so I'd start looking there.

On 8/9/2024 at 11:39 PM, ryan said:

MySQL can also support this with JSON encoded in a more traditional TEXT column with some reduced efficiency, though with the added benefit of supporting a FULLTEXT index. (Whereas the JSON column type does not support that type of index). For this reason, FieldtypeCustom supports both JSON and TEXT/MEDIUMTEXT/LONGTEXT column types. So you can choose whether you want to maximize the efficiency of column-level queries, or add the ability to perform text matching on all columns at once with a fulltext index.

Just a thought for future in case performance really becomes an issue for someone: it's possible to add a generated + stored column to a MySQL table that feeds itself from a dedicated value in a JSON column, and this column can of course have a fulltext index. A hook on PageFinder::getQueryUnknownField could then "redirect" the query to the generated column. Would only work on single valued fields, of course.

For anybody interested in creating such a generated field from a JSON field, here's a blog entry that explains it better than I could.

You should be able to append the required settings to $config->dbInitCommand.