Weekly update – 14 January 2022

[ryan]

This week we have some improvements to the $pages->findRaw() method. Most of these are based on requested improvements in either GitHub (like #427) or the forums:

Support for selecting parent properties/fields in the return value, i.e. 

$pages->findRaw("selector", "parent.title"); // Example 1
$pages->findRaw("selector", [ 'title', 'parent.title', 'parent.parent.title' ]); // Example 2

Support for selecting page 'meta' data in the return value, i.e. 

$pages->findRaw("selector", "meta"); // example 1
$pages->findRaw("selector", [ "meta.foo", "meta.bar" ]); // example 2

Support for selecting "references" in the return value. References are other pages that reference the found page(s) via Page reference fields, just like the $page->references() method. 

$pages->findRaw("selector", "references"); // example 1 
$pages->findRaw("selector", [ 'references.title', 'references.url' ]); // example 2

If you want the references to be indexed by field name, just include "references.field" in the requested fields:

$pages->findRaw("selector", [ 'references.field', 'references.title', 'references.url' ]); 

Support for selecting title and/or value from options (FieldtypeOptions) fields, i.e. 

$pages->findRaw("selector", [ 'my_options_field.*' ]); // example 1
$pages->findRaw("selector", [ 'my_options_field.title', 'my_options_field.value' ]); // example 2

Support for a new "flat" option that flattens multidimensional arrays in return value. This is best explained by an example. Let's say we did this:

$pages->findRaw("categories.count>0, limit=1", "title, categories.title" ); 

And it returns an array similar to this for every matching page: 

[
  'title' => '2-factor authentication coming to ProcessWire',
  'categories' => [
    1035 => [
      'title' => 'Security'
    ],
    1288 => [
      'title' => 'Users'
    ]
  ]
]

Now let's try the flat option:

$pages->findRaw("categories.count>0, limit=1", "title, categories.title", [ 'flat' => true ]);

(side note: you might already know this, but fields and options can also be bundled in the selector like this below, which is identical to the above):

$pages->findRaw("categories.count>0, fields=title|categories.title, flat=1");

It flattens any multi-dimensional arrays in the return value to be indexed in the same way that you requested them in the $fields argument:

[
  'title' => '2-factor authentication coming to ProcessWire',
  'categories.title' => [
    1035 => 'Security',
    1288 => 'Users'
  ]
]

In addition to these updates for the $pages->findRaw() method, this week there have also been updates to the $sanitizer->float() method and InputfieldFloat module. Now both support numbers with E notation such as 1E-2, 10E-14, -1.253E-5, etc. There are also some new values for the getString option on the float sanitizer. I don't work with this kind of stuff very often, so thanks to @MetaTunes on GitHub for helping to figure it out.

Lastly, the ProcessTemplate module also went through a refactoring with some minor improvements this week. Thanks for reading, have a great weekend! 
 

[kongondo]

16 minutes ago, ryan said:

Support for selecting parent properties/fields in the return value, i.e. 

Excellent! Thanks for this one! Thanks @dotnetic!

[Robin S]

@ryan, thanks for this week's updates!

@MoritzLost has sparked an interesting discussion about using PW with version control and he makes a comparison with Craft CMS. A couple of excerpts from his comments:

Quote

The problem for me is that too many areas of ProcessWire are not built with version control in mind. Migrations solve field/template configuration. But then you need to keep track of installed modules, if you want to install modules with Composer you need custom config so they're installed in the right folder. Of course you can't put the entire modules folder in gitignore, since site-specific modules go into the same folder. And what about ProcessWire itself, I don't want the entire core in my version control. So again, custom download script. The list goes on. You don't want to keep track of uploaded files, so the asset folder goes into gitignore. But I do want translations in my version control, turns out those are in random directories in the assets folder … the list goes on. There are just so many small issues if you want a 'complete' version controlled ProcessWire site, which to me means (1) the site can be rebuilt completely from the repository alone and (2) there's no external code or dependencies. And if you solve all of these with scripts, those become hard to manage and even harder to enforce in a team … not sure where I'm going with this to be honest ?

I'm a bit of an all-or-nothing person, if I can't have 'clean' version control I'd rather not have version control at all. Probably unwise. To be honest, for new projects I'm currently settled on option 3: to not use ProcessWire at all, unless the project won't require any significant updates after launch. Even all the problems mentioned above can be solved one way or another, it's just not worth it compared to other systems where I get full compatibility with version control by default. I don't think this will change as long as ProcessWire doesn't support version control natively.

Quote

Last year we've started using Craft CMS for some new projects, which has full support for version control out of the box. In terms of feature set it's quite similar to ProcessWire – you can define your own fields and entry types (similar to ProcessWire templates) and create custom field layouts. It's not a flat-file system, it does use a database. It also provides an API similar to ProcessWire's $pages to query for entries (pages) or other data. To keep track of schema changes, Craft saves all fields and entry type (and every other setting / config) as YAML files in a config folder. Those can be version controlled, merged and reasoned about easily. You can even read a pull request in Github and see what changes it introduces to field/entry type settings, it's great. It's called project config, here's the documentation. Particularly pleasant is that you can still use the backend to create fields and field layouts – every change in the backend is automatically written to the config folder. So you can just create field layouts in the backend and everything you need to track it in git is already there when you're done. To propagate changes between environments, the CLI has commands to apply the project config to the current environment (database).

I was thinking about writing a comparison between ProcessWire and Craft CMS regarding feature set, support for version control and development experiences. There are some good ideas there that ProcessWire could really benefit from.

Ryan, it would be great to get your voice in the discussion. We're at the beginning of the new year so what do you think about putting better compatibility with version control onto a roadmap for 2022?

[teppo]

Quote

But then you need to keep track of installed modules, if you want to install modules with Composer you need custom config so they're installed in the right folder. Of course you can't put the entire modules folder in gitignore, since site-specific modules go into the same folder. And what about ProcessWire itself, I don't want the entire core in my version control. So again, custom download script.

Very good points there and definitely a big thumbs up for this topic in general, but not sure if I can fully agree with this part:

• Any module that supports Hari's composer installer, Wireframe composer installer, or the official composer installers package will automatically be installed in the correct directory. Technically only requirements are that the module includes composer.json, and is marked as pw-module or processwire-module.
  • If you're using a custom directory structure (like we do), you'll indeed need a config setting in your main composer.json to specify where your modules directory is located — but even then it's a once-per-project thing. Hari's installer doesn't support this but others do, and Wireframe installer will also automatically detect any module's using Hari's installer and override it to make this work.
  • On a loosely related note, in case module authors are reading this: Hari's installer may still be the most popular option, but it's somewhat outdated. It lacks support for Composer 2, doesn't provide any configuration settings, etc. I would strongly recommend moving to more recent (and actively maintained) alternatives.
• I always add the modules directory to Git ignore, so it's definitely doable. This works out of the box for composer installed modules, and modules that are not installed via composer can still be added to the repository with the -f flag, which essentially makes Git ignore the ignore rule. Only problem would be modules added in a production environment via GUI, but since we're talking about version controlling stuff, this doesn't seem like a particularly relevant concern to me.
  • In case of modules that don't support Composer installation, creating a private fork is of course always an option. This works nicely for commercial modules as well, but requires installing said modules via Git repository (which is a bit clunky), or via private Packagist (which is more elegant, but requires paid plan.)
• In my setup ProcessWire is installed and updated via Composer. Again doable, but this part could admittedly be more elegant: since the wire directory needs to be in a publicly accessible location, typically you'll need post-install-cmd/post-update-cmd in your composer.json (or something similar) to move / copy said directory from vendor to another location.

Anyway, would definitely like to see this topic getting more attention in future releases. And it would of course be great if more module authors would add their modules to Packagist — that would make things easier for Composer users, and setting this up is quite straightforward. ?

In WP plugin installs via Composer are — for the most part — solved by wpackagist.org, which mirrors all plugins in the official plugins directory. In a nutshell this means that plugin authors just need to submit their plugins to the directory, and they automatically become installable via Composer (composer install wpackagist-plugin/plugin-name). Having something like that for PW would be an interesting option for sure.

[bernhard]

Side note @ryan it would be nice if you could add relevant keywords to the title of new news or blog posts. This makes it easier to find information when you are searching for it weeks or months later ("weekly update 14.1." is not nearly as helpful as "weekly update find raw parents, sanitizer float" in a search result)

+1 for better version control support. I've not built RockMigrations just for fun ?

[adrian]

Thanks @ryan - the "references" for findRaw is awesome. The only catch is that if there are no results, then we get an sql error.

Exception: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ')' at line 1 on line: 1311 in /var/www/myseatosky.org/html/wire/core/PagesRaw.php

The query it's complaining about is:

SELECT pages_id, data FROM field_actions WHERE data IN()

Let me know if you have any problem reproducing.

[adrian]

Sorry, also a bit confused about the results I am getting. Am I doing something wrong? Shouldn't I expect to see key / value pairs for title, and url for each of the results?

[ryan]

@adrian Thanks, I've pushed a fix for the IN() issue. For your other question, a blank 'references' array just means that there weren't any. Here's an example of what it looks like when there are some found (abbreviated print_r of the return value):

$pages->findRaw("template=category", [ 'references.field', 'references.url', 'references.title' ]);

Array
(
    [1017] => Array
        (
            [references] => Array
                (
                    [categories] => Array
                        (
                            [1061] => Array
                                (
                                    [url] => /pwsite/blog/posts/2-factor-authentication-coming-to-processwire/
                                    [title] => 2-factor authentication coming to ProcessWire
                                )

                            [1153] => Array
                                (
                                    [url] => /pwsite/blog/posts/processwire-3.0.109-adds-two-factor-authentication/
                                    [title] => ProcessWire 3.0.109 adds two-factor authentication
                                )
                                ...

without "references.field" it looks like this instead:

Array
(
    [1017] => Array
        (
            [references] => Array
                (
                    [1061] => Array
                        (
                            [url] => /pwsite/blog/posts/2-factor-authentication-coming-to-processwire/
                            [title] => 2-factor authentication coming to ProcessWire
                        )

                    [1153] => Array
                        (
                            [url] => /pwsite/blog/posts/processwire-3.0.109-adds-two-factor-authentication/
                            [title] => ProcessWire 3.0.109 adds two-factor authentication
                        )
                        ...

 

[dotnetic]

@ryan Thanks for the added supported fields in the findRaw method. I am still having performance issues with my queries, but I try to find the bottleneck or might even provide you with a dump, so you can recreate the issue. However great update, that will come in handy.

[adrian]

Thanks for the IN fix.

Sorry about the empty results issue - was a mistake in my selector.

Everything looks great and this is a really nice feature for generating long lists of documents, blog posts  etc associated with a person. Thank you!

 

[adrian]

@ryan - I wonder if it would be possible to also get the template name in the returned array? I need this to be able to group results.

I also need to the know the parent template name - could it be possible to have a sub array with details about the parent included for each result?

The other thing that I am struggling with is sorting the results. Obviously there are PHP ways to do this, but it would be great if PW's sort method was available here as well.

To give you a bigger picture, I am trying to convert:

$references = $page->references('has_parent!=1051, sort=template');
$references->sort('template, -date, -start_date, -year, -year_rt, -parent.month, title');

to use findRaw() with the new references option.

For this sort to work, I actually also need to be able to sort via the "month" field in the parent. 

I am guessing you'll say that this is beyond the scope of findRaw() which might be fair, but I thought I'd provide a real world use case where this takes the query time from:

1260.47 ms, 14.2 MB down to 297.23 ms, 854.6 KB so it's a huge performance gain if it can be made to work.

 

 

[gebeer]

On 1/15/2022 at 5:19 PM, bernhard said:

+1 for better version control support. I've not built RockMigrations just for fun

@ryanThis is a bit off topic, but I would also support version control for fields/templates as something to maybe concentrate on for this year. There is a lively discussion going on:

And Bernhard has already shown an awesome proof of concept in his video: