PW 3.0.193 – Core updates

[ryan]

ProcessWire 3.0.193 resolves 6 issues, makes improvements to the template and module editors, adds new hooks, adds improvements the $pages->find() findRaw method, and more. We covered some of these updates in last week's post, so we'll focus on what's new this week. 

First off we have a new advanced mode feature that lets you edit the raw configuration data for a module. This can be useful for various reasons, especially for module developers. If you have $config->advanced = true; in your /site/config.php file, you'll see a new option on your module information screen that enables you to directly edit the raw JSON configuration data for the module. 

There's also an option that lets you view the raw JSON module information data. Unlike the configuration data, this isn't editable. That's because it comes from the module directly (overtime you do a Modules > Refresh) or is generated at runtime, so there's little point in editing it here. 

In my case, I've found these new tools helpful for clearing out old and/or irrelevant configuration data during module development. In some cases, having the ability to edit this data may help to identify or fix issues that previously would have been difficult to do without using the API. If there's interest, I may move this into a dedicated (non-core) module that also lets you directly edit field and template configuration data too. But for now the feature is in the core, but just requires advanced mode before it appears. 

A few new hooks were added this week:

• Fieldgroups::fieldRemoved($fieldgroup, $field) 
  Called after a field has been removed from a fieldgroup/template.
   
• Fieldgroups::fieldAdded($fieldgroup, $field) 
  Called after a new field has been added to a fieldgroup/template.
   
• Fieldgroups::renameReady($template, $oldName, $newName) 
  Called before a fieldgroup is about to be renamed. 
   
• Fieldgroups::renamed($template, $oldName, $newName) 
  Called after a fieldgroup has been renamed.
   
• Templates::renameReady($template, $oldName, $newName) 
  Called before a template is about to be renamed.
   
• Templates::renamed($template, $oldName, $newName) 
  Called after a template has been renamed.
   
• Fields::renameReady($field, $oldName, $newName) 
  Called before a field is about to be renamed. 
   
• Fields::renamed($field, $oldName, $newName) 
  Called after a field has been renamed. 

These accompany the existing addReady(), added(), deleteReady(), deleted(), cloneReady(), cloned(), saveReady() and saved() hooks available for fields, templates and fieldgroups. 

Last week a couple people asked about versioning and migration of stuff in PW (like fields, templates, modules, etc.) and if there were any plans to provide additional tools for that. For the projects I work on at least, this part of the development process consumes so little time that it doesn't warrant developing more stuff for it. But I understand others might find it useful, so for those that would, I'd rather keep the core lean and instead leave that to tools/modules built by experts like Bernhard and others around here. 

I think it's important that whoever develops and maintains such features also be the same one(s) that would use them. But if any kind of core updates would be helpful to developers looking to implement more features here, I'm on board. Whether that means adding more hooks to specific events (see above as examples), maintaining field/template/module data in files in addition to the current DB tables, or anything else that helps such modules, this is all possible and likely simple for us to support in the core. So just let me know what I can do to help. 

While not full-featured migration tools, we do have useful field, template and page export/import tools in the core already, and those will of course continue to be maintained and improved, and may be expanded to include modules too. 

Thanks for reading and have a great weekend!

[wbmnfktr]

20 minutes ago, ryan said:

Last week a couple people asked about versioning and migration of stuff in PW (like fields, templates, modules, etc.) and if there were any plans to provide additional tools for that. For the projects I work on at least, this part of the development process consumes so little time that it doesn't warrant developing more stuff for it.

Would you mind telling us a bit more about your workflow in regards to projects and their migration?
This would be super interesting.

 

[Kiwi Chris]

Templates have a modified property, but fields don't. This might be a useful and simple core feature to implement that could make it easier for anyone wanting to track changes.

The Templates Export function shows the modified field, but it's sorted by template name. For quick manual exports, having the option to sort by modified timestamp would be an improvement.

Once fields have a modified property, they also need to have the option to have the export function list them by modified timestamp.

I see the base Wire class that everything else in the core extends has change tracking properties and methods, although currently there doesn't seem to be any built in method to enable persisting that with fields and templates when they're changed via the UI.

Up to date, I haven't really required the sort of version control others are discussing, but I have a couple of sites now, where I'm looking at bundling up functionality as modules for automated deployment to other sites, and the manual exporting of fields and templates doesn't really cut it, but I've built my data structures via the UI, so having an efficient way to convert that to code that can be used for multiple deployments would be useful.

With Visual Studio, I'm used to working with the option of using either visual designers or declarative code to build apps, and either way end up with code that can be included in version control.

 

[thetuningspoon]

I actually find that ProcessWire plays pretty well with Git, certainly in comparison to WordPress. The main thing is to avoid installing modules via the admin UI (just download the module and put it in your repo). And of course exclude /assets/ from the repo (PW conveniently keeps all of this together).

It would be nice to be able to exclude the wire folder and have Composer handle that instead, but I don't find it that big of a deal to keep the wire folder in Git. In the very rare case that a direct core mod is necessary for a project I'm working on, it's good to be able to make that change and commit it to the repo.

The ability to maintain a master configuration file containing all of the meta data for the templates and fields in a project and to be able to put this into version control would be a real game changer, though. This is a real pain point we're running into on a lot of our projects, especially now that we have a couple systems that have multiple deployments and multiple developers working on them.

Since we already have JSON export for templates and most field types (the options field export/import is still not fully functional), it doesn't seem like this is too far off. Maybe PW could cache a copy of the JSON configuration file and then if it gets updated externally, show a message in the admin that there are pending field/template changes to be applied, allowing the user to apply these changes with a single click. If it makes it easier/cleaner, there could be 2 config files, one for fields and one for templates (and maybe another for modules?)  I'm curious how other systems handle this under the hood.

Edit: Here's how Craft CMS does it: https://craftcms.com/docs/3.x/project-config.html#propagating-changes  This sounds a lot like what I'm thinking. The command line option for applying changes is also a great idea since some changes could take a while to apply on large projects.

[bernhard]

41 minutes ago, thetuningspoon said:

This is a real pain point we're running into on a lot of our projects, especially now that we have a couple systems that have multiple deployments and multiple developers working on them.

Ever tried RockMigrations?

[ryan]

Quote

Would you mind telling us a bit more about your workflow in regards to projects and their migration?

I use the tools that ProcessWire comes with. I spent a lot of time making them simple and easy to use, and that's what I like to use. Just as an example, let's say that I've got a website and a client wants to add a full featured blog to it. I'll develop it on my local copy of the site and it might involve creating several fields, templates and template files. I'll take a day or two to develop it and when it comes time to migrate the finished work to the live server, that's the fun part, but there's not much to it:

1. Create or export/import the new fields on the live site first, then do the same for the new templates.
2. Copy the new or updated template files (and related CSS/JS assets) into place.
3. Create or export/import the pages needed by the blog, and it's done.

A blog is just an example but it's the same as any other update. It's a painless process that always goes quickly and smoothly. This part of it takes maybe 5 to 10 minutes and is one of my favorite parts of the project, like driving a new car and then seeing it for the first time in your driveway. I like to oversee this part of any project and have no need to optimize it further so guess I'm not the target market for add on migration tools. 

Quote

Since we already have JSON export for templates and most field types (the options field export/import is still not fully functional), it doesn't seem like this is too far off. 

That's correct, it would be fairly straightforward.

 

[thetuningspoon]

On 1/25/2022 at 2:11 PM, bernhard said:

Ever tried RockMigrations?

I looked into it a while back but wasn't sure how to use it. I think I need a little tutorial that walks though a particular use case.

[bernhard]

Thx for the feedback. I guess many who are having your painpoint are missing RockMigrations because of this reason...

I've updated the first post of RockMigrations with an easy step-by-step guide:

 

[thetuningspoon]

Thanks @bernhard, that's helpful. I'm still not sure how the typical workflow would look, though. Would you create a new file for each set of changes that you make to your site? Or do you just keep updating that one array that defines all of the templates and fields for the whole site (like a declarative configuration file)?

 

[Kiwi Chris]

I've found what seems to be an undocumented method of the Field class that's used by the built in field import/export that I think could be useful in automation.

$field->getExportData() returns details of the object, but needs a little bit more work to turn into a JSON object, but builtin ProcessFieldExportImport shows how to do that.

There's a corresponding $template->getExportData() method.

I tried this:

$testField = $fields->get('headerImage');
echo wireEncodeJSON($testField->getExportData(),true,true);

and I got a nice JSON object with the field definition in return.

I'm lazy, and forgetful, so I like using the admin UI to build my fields and templates, because it reminds me of all the properties of a field and what they do, right there on screen.

Because I'm forgetful, where the UI gets a bit less useful is exporting fields, as I've got to try to remember which ones I changed. Adding a modified property and sorting by this as I previously suggested would help, but if I want to deploy the same configuration to a bunch of sites, manual export/import isn't really scalable.

With this, I can potentially be lazy and forgetful, as I don't care how a field is made, I can simply grab it's definition and write to a JSON file, but I don't have to try to remember what fields are required for specific functionality as I can simply provide an array with a list of fields (and templates) required in a module, and let it generate them from the JSON files.

I think what would be useful at the core level would be proper documentation for these import/export methods that already exist, and maybe a wrapper function so that it's possible to get directly as JSON data rather than requiring an extra step. eg $field->getJsonData() or something like that.

I'm sure these JSON files could also provide field and template definitions for Bernhard's RockMigrations.

[adrian]

2 hours ago, Kiwi Chris said:

maybe a wrapper function so that it's possible to get directly as JSON data

You can copy it directly from Tracy when viewing a field's settings. Does that help?

[wbmnfktr]

On 1/26/2022 at 9:56 PM, ryan said:

I use the tools that ProcessWire comes with.

Thank you so much for this insight... even though I hoped for something completely different - to be honest. 
I thought you would use some database-magic-scripts or an individual what-has-changed-since script, but I'm totally fine and impressed with your (don't know how to say, but...) focus and clear view about all the things you changed from one dev-step to another. I personally can't remember one step after another even though I write almost everything down (no, just kidding. It's not that bad! But close.).

There is one thing in your routine that I can't work around...
How do you manage conflicts when creating new templates, withouth fields that haven't been imported/created, yet.

OR...

How do you manage conflicts when importing fields, when templates those fields belong to don't exist?

To be fair and honest again... I almost never really used those export/import features. Even though I always wanted... but... other ways worked way better, faster, and way easier for me in those moments.

When this/that/these/those... tools you use work for you... I might have to look way deeper into it, than I ever thought. Or at least make me familiar with them.

WOW... 

 

 

[thetuningspoon]

If I get some time I would love to work on building a module that would deliver a Craft CMS style solution for this, since I think that would work best for our projects. Just not sure whether I’ll get that chance or not.

[wbmnfktr]

On 1/25/2022 at 8:11 PM, bernhard said:

Ever tried RockMigrations?

I did, this week.

Not for the very first time, but for the very first time within a very real project.

It worked. Until it became complicated in some ways and I had to change, switch, move, re-add, and so on.
It took me about 2 hours to get things done I normally do within 10 minutes. Yet... that was on purpose. I wanted to try it again but on the other hand I have had a time-limit.

I guess RockMigration can do this, but I don't know how yet, but... how could I export an existing set of fields, templates, individual settings from an existing setup into an RM-script?

My favourite goal would be... installing RM into a lot of existing projects, create RM-export scripts that put everything into a PHP (or whatever) files I can use later on to re-import/re-migrate into a new project. So I could build up some kind of library for cases I always need.

Is there a RM-function that allows me to export all fields, templates, and field/template settings into an RM-PHP?

In case this sounds weird, I'd understand. So... maybe... just drop me a message with "You are weird!" better though... a message with "Yes... (but)... look here... use this function here and there".

 

I somehow start to understand this module. Even though it's way out of my comfort zone.

[wbmnfktr]

2 minutes ago, thetuningspoon said:

If I get some time I would love to work on building a module that would deliver a Craft CMS style solution for this, since I think that would work best for our projects. Just not sure whether I’ll get that chance or not.

Craft CMS is on my list for 2022, but don't tell anyone.
Yet... I'd suggest talking to at least with @bernhard.
I think he is soooo close to a solution we all have (somehow) in mind.
Yet maybe we don't understand his RockMigration right now.

[horst]

18 minutes ago, wbmnfktr said:

I might have to look way deeper into it, than I ever thought. Or at least make me familiar with them.

You definetely should! ?

Have a look at the Screenshots and its 100% save to: 

On 1/3/2022 at 5:14 PM, horst said:

PS: * In case you are not sure if you have written everything down on paper, you can generously include additional fields or templates in the export. In this case, it is better to have more (or even all) than too few. All unchanged settings will be recognized by PW during import and will not be processed further. ? 

 

[horst]

25 minutes ago, wbmnfktr said:

There is one thing in your routine that I can't work around...
How do you manage conflicts when creating new templates, withouth fields that haven't been imported/created, yet.

FIELDS first!
TEMPLATES second!
CONTENT last!

Even if you want add two new templates at once with a family relation like parent <>child, you paste in the JSON on the import site, it tells you about the conflict after inspecting. The solution is simply to run the import twice (two times). That is already supported by the importer.

 

[wbmnfktr]

13 minutes ago, horst said:

You definetely should! ?

Sure... I know your workflow... yet... whenever I try it... it's like some illegal firework blew right into my face and my weekend was ruined.
It's probably me in some kind or another.

It's a workflow that doesn't fit into mine for some kind or another for whatever reason.
Maybe I'm too old for those things or maybe I worked way too long with exports that mirrored all changes far too perfect from one system to another. (INXIRE CMS on Oracle - in case you want to know).

But SURE I will look into this more but on the other hand the CraftCMS and CSV/JSON-Recorder I saw was almost perfect for my cases.

[wbmnfktr]

3 minutes ago, horst said:

FIELDS first!
TEMPLATES second!
CONTENT last!

That sounds solid and reliable.

4 minutes ago, horst said:

Even if you want add two new templates at once with a family relation like parent <>child, you paste in the JSON on the import site, it tells you about the conflict after inspecting. The solution is simply to run the import twice (two times).

I'd almost bet on this but it never worked out somehow. But could have been right after this feature was introduced. Can't say for sure. Never tried it since or at least for a long time.

[Kiwi Chris]

1 hour ago, adrian said:

You can copy it directly from Tracy when viewing a field's settings. Does that help?

Tracy was useful in testing out what I'm trying to achieve. I'm trying to get away from copying and pasting though.

I used the Tracy console, and this code is on the track of what I want:

$modulePath = $config->path('MyModule');

//moduleFields is an array of field names
foreach($moduleFields as $fieldname){
    $testField = $fields->get($fieldname);
    $data = wireEncodeJSON($testField->getExportData(),true,true);
    $file = $modulePath . 'data/fields/'. $fieldname . '.json';
    file_put_contents($file, $data);
}

This gives me a nice directory of field definitions that are required by a given module.

ProcessWire already handles module dependencies including module versions with the requires property in getModuleInfo(), so if I define fields that are used in multiple places in a base module and specify that as a dependency of other modules that install other fields, then I can make sure I end up with all the fields that I need. Because I'm using a manually created array of field names, if there are fields that depend on other fields, I can list them in the right order to satisfy dependencies.

Fields like page reference fields can depend on a template as well, so my array might need to specify both object name and type to install all the dependencies in the correct order, but I think I'm on track to being able to build a reproducible configuration I can easily deploy to multiple sites. 

I think it might not be too hard to build and admin module to provide a UI to build the dependency configuration files for custom modules either, as ProcessWire already supports adding and reordering items in lists in various Process modules.

Edit:

The reason I'm outputting one JSON file per object rather than just one big JSON file as the built in exporter gives you if you select multiple fields, is that I think it will be better for source version control, especially if a modified timestamp property is implemented for fields, and if there's a need for multiple collaborators on a project. Unmodified object definitions don't need to be regenerated.

Additional Edit: Usually a module provides some kind of functionality, but there's no reason why a module can't just exist to install/uninstall a bunch of fields and templates, and otherwise just sit dormant. If it's not an autoload module, it's not going to do anything unless it's called from somewhere, and a module seems like a good way to encapsulate a whole lot of stuff. With the right design around dependencies and inheritance, it should be possible to change 'skins' or 'themes' for a site; something that's not possible with site profiles, as they're basically just a dump of a specific site state. eg, you could have baseModule that installs xyz fields and templates, then themeModule1 and themeModule2 that each provide different front end presentation, but depend on baseModule, so there can be certainty that all the fields they need exist.

[wbmnfktr]

In summary... @ryan and @horst use what PW provides... @bernhard uses his RockMigration... all others of us use what ever fits our need and I'm totally fine with that... and really appreciate that.

Yet... somehow I'd really love to see a CraftCMS-like (even though I never saw or used it) JSON/YAML-Recorder-style tool @bernhard showed to us.

What if we open a money-pool for such a development?
Someone interested?

If I could have some kind of a solid working solution for a real time JSON/YAML-file-export/import solution which I only need to migrate via Git, without writing any functions, code or whatever, solely based on my PW-setup, to make all my instances the same... sure with some checks in the background but... YES... that would be my dream.

Still... I want and have to try and test the solution CraftCMS has available.

I put my money where my mouth is and start upfront with EUR 500 in the pool.

I will post some details of all the features I'd like to have and see in such a module in the upcoming days, maybe a week or two.
Yet... whoever wants to join, let me know.
In case you want to develop such a module, PLEASE let me know.

I guess I want to see this module happen in some kind or another (community or pro module).

My goal is... making PW even better. And maybe this could be another step.

 

This thought elsewhere (Added: 2022-01-29):

 

[AndZyk]

TBH I totally forgot about the Export/Import feature of templates and fields until I have read about it here again.

My workflow was to have two windows open (developement and staging/production) and manually rebuild fields I have created in my development environment.

 

But export/import is way easier. Thank you for reminding me. Maybe this feature should be more prominent with additional buttons at the beginning of the overview for people like me who don't look at the end of the page. ?

As for the other discussed solutions: A JSON/YAML solution would also be great for version control, but for me it would not be necessary, because in my experience the gap between development and staging/production is not that large. But for large websites this could be handy.

[bernhard]

18 hours ago, thetuningspoon said:

I'm still not sure how the typical workflow would look, though. Would you create a new file for each set of changes that you make to your site? Or do you just keep updating that one array that defines all of the templates and fields for the whole site (like a declarative configuration file)?

The latter one. The very first two versions used the former option, but it turned out it is hard to maintain and even harder to grasp. Using config based migrations it's simply copy&pasting field config data from tracy to a migration file and you instantly see what the migration does (and you get a diff view via git).

15 hours ago, Kiwi Chris said:

I'm sure these JSON files could also provide field and template definitions for Bernhard's RockMigrations.

Correct

11 hours ago, Kiwi Chris said:

//moduleFields is an array of field names

Where does this array come from?

11 hours ago, Kiwi Chris said:

Additional Edit: Usually a module provides some kind of functionality, but there's no reason why a module can't just exist to install/uninstall a bunch of fields and templates, and otherwise just sit dormant. If it's not an autoload module, it's not going to do anything unless it's called from somewhere, and a module seems like a good way to encapsulate a whole lot of stuff. With the right design around dependencies and inheritance, it should be possible to change 'skins' or 'themes' for a site; something that's not possible with site profiles, as they're basically just a dump of a specific site state. eg, you could have baseModule that installs xyz fields and templates, then themeModule1 and themeModule2 that each provide different front end presentation, but depend on baseModule, so there can be certainty that all the fields they need exist.

This sounds a bit like you are starting to understand RockMigrations... ? 

12 hours ago, wbmnfktr said:

It worked. Until it became complicated in some ways and I had to change, switch, move, re-add, and so on.

It would be nice if you could provide more details on that point. What did you do? How did you do it? What did you try to do? etc

[Kiwi Chris]

7 hours ago, bernhard said:

This sounds a bit like you are starting to understand RockMigrations... ? 

The concept of RockMigrations makes a lot of sense, but what seemed to make it hard, was it depended on manually building object definitions for migrations in code.

It turns out there are ProcessWire core methods to export field, template, and page definitions as JSON, so that makes it a lot less code to write.

Where I'd like to get to is have a list of field, template, and page dependencies in a module configuration, and be able to do something like $module->build(); to generate JSON files for all those dependencies in the module directory.

Modules already have install() , uninstall() and upgrade() methods so they could look for any object dependencies in the module config and create (or remove) them as required.

9 hours ago, bernhard said:

Where does this array come from?

Here's a working proof of concept I've been playing with in Tracy console.

Although I prefer to use the UI when I can, due to the issues around dependencies between fields, templates, and pages, manually specifying an array with the objects in the order they need to be created avoids the issues others have run into of having to make multiple passes to satisfy all dependencies.

I could make it smarter by checking the modified timestamp for templates and pages (and hopefully soon fields if Ryan adds it to the core), to only generate new JSON files for objects that have changed since the last build.

$requiredObjects = array(
							'competitionImage' => 'field',
							'compGrade' => 'field',
							'competition' => 'template',
							'compId' => 'field',
							'competitionCalendar' =>'template',
							'competitiontopics' => 'page',
							'media' => 'page',
						);

$modulePath = $config->path('MyModule');

$pageExporter = new PagesExportImport();
foreach($requiredObjects as $objectName => $objectType){
    switch($objectType) {
        case 'field':
            $object = $fields->get($objectName);
        break;
        case 'template':
            $object = $templates->get($objectName);
            break;
        case 'page':
            $object = $pages->find("name={$objectName}, include=hidden");
            break;
    }
    if($object){
        $file = $modulePath . "data/{$objectType}s/". $objectName . '.json';
        if($objectType != 'page'){
            $data = wireEncodeJSON($object->getExportData(),true,true);
        }else{
            $data = $pageExporter->exportJSON($object);
        }
        file_put_contents($file, $data);
    }else{
		//Object doesn't exist!
        echo $objectName;
    }
}

 

[bernhard]

3 minutes ago, Kiwi Chris said:

manually specifying an array with the objects in the order they need to be created avoids the issues others have run into of having to make multiple passes to satisfy all dependencies.

RockMigrations simply creates the fields and templates first and after that populates their settings. That makes it possible to use things like parent child relationships in one single migrate() call.

11 minutes ago, Kiwi Chris said:

The concept of RockMigrations makes a lot of sense, but what seemed to make it hard, was it depended on manually building object definitions for migrations in code.

I have thought about that a lot (meaning years) and still have not found a way how one could reliably record and then migrate stuff without writing code and without telling ProcessWire what to do exactly. Writing the setup to files is one thing (that's basically as easy as a foreach + export()), but migrating those setups is another. I know such recorders are tempting, but everybody that has ever used the excel macro recorder knows what I'm afraid of... If you have good ideas how to solve that please let me know.