Provide a way to define / save / sync fields in the code base

[Rasso]

Dear ProcessWire folks!

First of all, thanks for all your work on this amazing CMS @ryan and contributors! ❤️

I'm a WordPress guy with 10+ years experience. I'm well aware of the shortcomings of WP (chaos API, no built-in multilingual support, to name a few), but was always able to find a way to work around these. In 2023, due to the never-ending praise from some of my developer peers, I finally gave ProcessWire (and also Kirby CMS) a try for two small projects. If nothing else, these excursions gave me a more senior perspective on the advantages and shortcomings of these three content management systems (none of them is perfect).

I was impressed by many thoughtful decisions in ProcessWire's setup, data structure and API. But I am missing one last thing that both WordPress and Kirby have solved in a very clean an comfortable way:

A clean solution for deploying fields between environments and committing them to version control.

• ✅ In WordPress, I am using Advanced Custom Field's local JSON feature for this
• ✅ In Kirby, all fields and templates are being defined using YAML blueprints, so they are also easy to deploy and version control (although YAML configs come with their own set of shortcomings ?)
• ❌ In ProcessWire, I couldn't work out a reliable way to define (/autosave/sync) the fields in my code base. 

So, my question is: Is there any chance the core team will give first-class support for programmatic deployment of fields another thought? If not, what are the considerations that lead to this decision? Syncing fields manually between environments is too cumbersome and error-prone for my taste.

I know my request is not new, but the information available felt a bit outdated and not very convincing to me. I'd like to start a discussion here based on facts, not opinions.

[Rasso]

To add a bit more information: In the website I built with PW, I was using the Pro field Repeater Matrix. I tried RockMigrations (as the original Migrations module has been deprecated) with that field and it didn't work (can't remember what exactly went wrong, sorry for not documenting my findings better).  

I would be very open to pay for a new module Pro Migrations by the core team of ProcessWire, if that could provide a robust experience that would integrate with all pro fields.

[d'Hinnisdaël]

Welcome @Rasso!

While ProcessWire doesn't have built-in support for migrations or version-controlled field definitions, it already supports export and import of field definitions via JSON, albeit only manually via the admin interface. However, it being JSON-based should be a good enough base to start automating some of this, e.g. by saving and reading the JSON from disk. That would also make them version-controlled as well.

I agree this should be part of the core. Most CMS have solutions for this and it'd be a useful feature to advertise. The landscape of modules here is somewhat splintered. The module that integrates best with the core, Migrations, has unfortunately been deprecated. I haven't fully tested RockMigrations yet, but from what I've seen it seems to be doing a few things too many (snippets, "magic pages", "files on demand") and I'd prefer a module that limits itself to migrations, if not just for performance.

+1 for a ProMigrations module from @ryan to ensure the best level of integration into the core.

[MarkE]

9 hours ago, d'Hinnisdaël said:

However, it being JSON-based should be a good enough base to start automating some of this, e.g. by saving and reading the JSON from disk.

I have a json based migrations module that I released some time ago (ProcessDbMigrate). It needed further work, so I have not publicised it further after the initial proof of concept. In the meantime I have improved it enormously and tested it quite extensively with multiple field types including RepeaterMatrix. It is almost ready for re-release. It is a completely different concept to RockMigrations, which is an excellent and well-established module. My module will automatically track database changes as you make them in the back end UI and then export json for installation in the target. It also provides for roll-back, database comparisons and much more. Hopefully out before Christmas. 
 

PS I found during the course of development that there are quite a few flaws in the native export functions and had to re-write quite a bit. 

[d'Hinnisdaël]

@MarkE Sounds interesting! Keep us posted about new releases and let us know if you need beta testers.

[da²]

7 hours ago, MarkE said:

I have a json based migrations module that I released some time ago (ProcessDbMigrate).

Did you ever think of using fields and templates data directly from database? ("fields" and "templates" tables)

On my current project, while the site is still in closed beta I just push my dev database to remote server, so updating is easy. But a time will come when I'll have to replicate my changes without erasing production site. ? I think about testing RockMigrations but I'm also thinking about something more targeted to my main needs: most of the time I'll just add and configure new/existing fields and templates, and it looks like it can be done with "fields" and "templates" database tables.

I don't know how ProcessDbMigrate works, I looked the code very quickly and think you parse PW data using API, like RockMigration does if I'm right. Dunno if directly using the database is a good idea, but that looks faster to implement and more robust to me actually because you don't need to take care of future PW features or changes to add them in your JSON exporter/importer.

Just sharing some thoughts...

[Rasso]

@MarkE that sounds very promising to me! Have you thought about reaching out to @ryan about possibly merging your module into core and/or work on the functionality together? That's how I know it from WordPress – new feature proposals are often first being developed in so-called "Feature Plugins", and merged into the core as soon as they are ready for prime time.

In my opinion, the problem with using modules for central functionality like this is, that they might become deprecated, as it happened with the original Migrations module. Also, you would have to carry the burden of making sure that the module would continue to work with the ProcessWire field types. That's why I'd always prefer functionality like this to be as close to the core as possible. Either in core or as a Pro plugin.

Quote

PS I found during the course of development that there are quite a few flaws in the native export functions and had to re-write quite a bit. 

This definitely sounds like something I'd reach out to Ryan about. You probably know the saying... "If you want to go fast, go alone. If you want to go far, go together" ?‍♂️

[MarkE]

11 hours ago, d'Hinnisdaël said:

Keep us posted about new releases

Will do. Please note that the code on GitHub may be behind the latest version and may well contain bugs. I will post on the Forum when I am happy with it.

4 hours ago, da² said:

most of the time I'll just add and configure new/existing fields and templates, and it looks like it can be done with "fields" and "templates" database tables

I haven't tried this. It might work if you have not changed any pages at all. I would be afraid of breaking relationships and, personally, I sometimes use pages to hold settings that are not user-updatable and which I might need to migrate, so that would not be a solution for me.

4 hours ago, da² said:

you don't need to take care of future PW features or changes to add them in your JSON exporter/importer

Fair point. Particularly re any special field types introduced in modules. ProcessDbMigrate does cater for most of them (I am just updating it for FieldtypePageTable) including some which store attributes as objects, but would need updating if there are major changes or additions to the core (my guess, though, is that these are very unlikely to be breaking changes).

1 hour ago, Rasso said:

Have you thought about reaching out to @ryan about possibly merging your module into core and/or work on the functionality together?

It is clear to me from previous discusssions that @ryanis not convinced of the need for this in the core and, indeed, does not feel the need for it personally. I consider that a shame as the lack of this functionality is in my view the only major missing feature in ProcessWire. The topic has been debated here numerous times with no clear conclusion. As a consequence, those of us who feel the need have had to turn to modules - either our own or others'. RockMigrations has been released for a while and is well-supported. Both that and ProcessDbMigrate are open-source so, unless @ryanchanges his mind, it will be up to the community to contribute to module development or, indeed, take it over should the original author no longer maintain it. That will depend on how important the community thinks it is.

1 hour ago, Rasso said:

This definitely sounds like something I'd reach out to Ryan about.

I think I reported some issues without any action being taken. Also, I recall that the export functions were left as work in progress some time ago. But I may mis-remember...

[dotnetic]

Welcome to our lovely community, Rasso. I hope you are enjoying the benefits of ProcessWire as much as I do.

We had the discussion about replicating / syncing fields on the server many times. I use RockMigrations as a reliable and working module for this.

You write code with migrations and they get executed as soon as you save the file and refresh the admin.

A migration can be inside the site/migrate.php file, or in a migrate method inside of your custom module, or somewhere else.

Here is a possible example of a migration:

$rm->migrate([
    'fields' => [
        'body' => [
            'label' => 'Content',
            'type' => 'FieldtypeTextareaLanguage',
            'inputfieldClass' => 'InputfieldTinyMCE',
            'tags' => 'generic',
            'contentType' => 1,
            'langBlankInherit' => 1,
            'inlineMode' => 0,
            'height' => 500,
            'lazyMode' => 1,
            'features' => [
                0 => 'toolbar',
                1 => 'menubar',
                2 => 'stickybars',
                3 => 'spellcheck',
                4 => 'purifier',
                7 => 'pasteFilter',
            ],
            'toolbar' => 'styles bold italic pwlink pwimage blockquote hr bullist numlist anchor code',
            'rows' => 15,
        ],
        'home_intro' => [
            'label' => 'Home Intro',
            'type' => 'textarealanguage',
            'tags' => 'generic',
            'inputfieldClass' => 'InputfieldTinyMCE',
            'rows' => 5,
            'inlineMode' => 1,
        ],
        'subheadline' => [
            'type' => 'text',
            'inputfieldClass' => 'InputfieldTextLanguage',
            'maxlength' => 100,
            'tags' => 'generic',
        ],
],
'templates' => [
        'basic-page' => [
            'label' => 'Standardseite',
            'fields' => [
                'title',
                'navtitle',
                'seo',
            ],
        ],
        'imprint' => [
            'label' => 'Impressum',
            'fields' => [
                'title',
                'body',
                'seo',
            ],
        ],
]
]);

With the migration you can also create pages and even fill them with content (if you want that)

$rm->createPage(title: 'About us', name: 'about-us', template: 'basic-page', parent: '/');

There is no sync between fields or templates you create in the admin and RockMigrations (there was a YAML option once, but I don't know if it still exists), but you get an easy to copy code in each fields edit section, that you can copy directly over to your migration.

I am also in favor that migrations should be a core feature of ProcessWire, but Ryan does not approve. So we have to stick with what we have, and RockMigrations is a great solution.

[szabesz]

5 hours ago, MarkE said:

It is clear to me from previous discussions that @ryanis not convinced of the need for this in the core and, indeed, does not feel the need for it personally. I consider that a shame as the lack of this functionality is in my view the only major missing feature in ProcessWire. The topic has been debated here numerous times with no clear conclusion.

I can confirm all this... and I also in favor of a solution that works the way @MarkE designs it. As soon as it is ready for testing and I have the time, I will surely try it out as well.

[MarkE]

On 11/27/2023 at 12:14 PM, d'Hinnisdaël said:

being JSON-based should be a good enough base

One thing I should mention (and would appreciate some feedback on) is that my migrations module (and I guess any that uses json files) is declarative. Even if you create a migration automatically while making your database changes, what happens is the module logs what objects have changed and exports the final state. It does not operate as a 'macro' logging each change separately. The disadvantage of this is that problems might arise where there is a 'dependency cycle' - e.g. a new field (say a page ref type) depends on a new template which includes the field. With my module, if this happens while you are logging changes it will (should ?) warn you. If you create the migration manually and then 'sort on save' to get the items in dependency order, it will give you an error message (see video at end).

I think there are 3 ways of dealing with this:

1. Do more than one migration, so that all required objects are present in the database.
2. Include an item twice in the migration - firstly without the dependency, then with the dependency after the other object has been added.
3. Install the migration twice and hope it will sort itself out (i.e. the misssing objects will be there on the second installation).

My module encourages (1) but permits (3) - as you will see. It does not attempt (2) - this might be feasible by detecting the cycle, and breaking it for the first item, but a bit complex to implement.

I'd be interested in any thoughts people have on this issue (and indeed on how RockMigrations deals with it - @bernhard?).

Meanwhile, to amuse you (??) here is a video of a big complex migration I did as part of the testing process. As you will see, there are lots of cycles, but it installed with just two clicks of the 'install' button ?. 

 

 

[szabesz]

@MarkE I rephrased your post above a little bit and fed GTP-4 with it. This is what I got:

Circular dependencies can indeed present a challenge when dealing with database migrations. Your proposed solutions are viable and could work depending on the specific situation. Let's delve deeper into each approach and discuss their potential benefits and drawbacks:

1. Performing the database migration in multiple steps: This approach is generally safe and effective, as it ensures that all required objects are present in the database in the end. However, it may require careful planning and sequencing of migrations to ensure that each step has all its dependencies satisfied. This approach is encouraged by your module and is also recommended in Django's migration documentation [Source 2](https://django.readthedocs.io/en/stable/topics/migrations.html).

2. Including an item twice in the migration: This approach involves initially adding the class or field without the dependency, then adding the dependency after the referenced object has been added. This method can work, but it may be complex to implement and maintain, especially for large databases with many interdependencies. Your module currently doesn't support this approach.

3. Running the migration process twice: This approach relies on the hope that the second run will resolve any missing dependencies from the first run. While this may work in some cases, it's not a guaranteed solution and could potentially lead to inconsistent results. Therefore, it's typically not recommended unless you have a strong understanding of the specific dependencies and migration behavior in your database.

Another approach worth considering is the use of **"swappable" dependencies**, as mentioned in Django's migration documentation [Source 2](https://django.readthedocs.io/en/stable/topics/migrations.html). This allows you to declare dependencies that can be swapped out with different implementations, which can be useful when dealing with circular dependencies.

Hmmm....

 

[gebeer]

On 11/27/2023 at 7:00 PM, Rasso said:

I was using the Pro field Repeater Matrix. I tried RockMigrations (as the original Migrations module has been deprecated) with that field and it didn't work (can't remember what exactly went wrong, sorry for not documenting my findings better).

Which version of RockMigrations did you use? I added Repeater Matrix support and this was merged approx. 2 months ago. Here's how to use it: https://github.com/baumrock/RockMigrations#repeatermatrix-field

Repeater Matrix support is also mentioned on the module's page https://processwire.com/modules/rock-migrations/

If you tried the version with Repeater Matrix support and it didn't work for you, please let us know what exactly went wrong so we can help to fix it. In all my tests and using it in a production project, I didn't encounter problems so far.

RockMigrations has been a big time saver and working reliably for me for the past 2 years or so. I haven't tried other solutions mentioned here since they are not actively maintained.

As for having this functionality in the core. Yes, that would be great, indeed. Honestly, I don't understand, why @ryan hasn't implemented it yet. Maybe also because there is a great solution with a well designed API available already as a free module? Still, I think it would benefit the project if this was either in core or available as an optional core module. IMO, migrations are essential, especially when using PW for bigger projects and when working on projects in a team.

As for the discussion here about potential problems with migrations brought up by @MarkE , I was able to work around all of those with RockMigrations since it is also declarative, but not destructive (unless explicitly intended).

On 11/27/2023 at 7:14 PM, d'Hinnisdaël said:

I haven't fully tested RockMigrations yet, but from what I've seen it seems to be doing a few things too many (snippets, "magic pages", "files on demand") and I'd prefer a module that limits itself to migrations, if not just for performance.

The "few things too many" argument is flawed since all the extra functionality is optional and therefore doesn't impact performance. And the addon-features are well designed and can be very useful. Performance with RockMigrations is really good. Only very large migrations take some time. I have a project with about 4000 lines of migrations for templates, fields, pages and roles in my main migration file and then some additional 1000 lines spread throughout modules and these take about 20 seconds. Since you are not doing migrations all too often, this is very tolerable. Migrations on smaller projects are almost not noticable.

[bernhard]

Glad to hear some facts from someone actually using my module and not just guessing ? 

11 hours ago, gebeer said:

I have a project with about 4000 lines of migrations for templates, fields, pages and roles in my main migration file and then some additional 1000 lines spread throughout modules and these take about 20 seconds. Since you are not doing migrations all too often, this is very tolerable. Migrations on smaller projects are almost not noticable.

I want to add here that during development it's never ever 20s for me and I'm using migrations all over and all my projects would simply not be doable or manageable without the module. The reason is that I've split migrations into smaller pieces rather than having them in one huge file. If you do that, RockMigrations will take care of only migrating the file that has actually been changed, so migrations during local development are a matter of milliseconds.

For example I'd have a FooPage.php custom pageclass and there I'd add migrations that do stuff related to the FooPage, for example adding foo_field_body and foo_field_description. One of these "few things too many" aka MagicPages takes care of triggering the init() and ready() method of this pageclass, so that I can watch these files and trigger the migrate() method whenever the file is saved. I'd prefer if that was a core feature, but it is not.

On deployment RM will run all migrations and that might take a little. But I don't know how long exactly because Github does that for me with the help of RM's deployment tools. Usually a full deployment with copying files etc. takes about a minute.

Let's say that's a client project and the client contacts me 2 months later that something doesn't work. What I'd do is to execute "rockshell db:pull production" and some seconds later I'd have the current state of the project with all the new data on my local environment. While browsing the new content one of these "few things too many" aka filesOnDemand takes care of downloading all the images that the user has uploaded in the meantime that were not transferred by the db:pull. It does that in the background and only if I want it and have $config->filesOnDemand = ... in my config file.

Does RockMigrations have a fancy "revert" button? No. Is it possible to revert changes? Yes! If you need that, do it. Just write the according revert migrations and trigger them. I don't do that for every migration, because I don't want to do work that I never ever need. If I change my mind during development and want to go another route all I need is "rockshell db:pull" and all changes are "reverted". Sometimes (in maybe 0.5% of all migrations) it still happens that I want to revert changes that I've already pushed to production. What I do in that case is to add some "cleanup" (you could also call it "revert") migrations at the top of the actual migrate() method. Something like $rm->deleteField('fooclass_bodyfield', true).

You could even remove that line of code later, once that change has been applied to production. "Waaaah, you can't do that! What if others are working on the same codebase and never triggered that migration?" I hear you say... Well, I can. Everybody working on my codebase has to do two things before starting to work:

1) "git pull" to get the latest code state
2) "rockshell db:pull" to get the latest database state

So there will not be any "fooclass_bodyfield" for him/her and therefore it's fine to not have the deleteField() call in their migrations.

Another benefit of splitting migrations into pieces is best shown by RockPageBuilder. There all blocks have their dedicated migrations. A slider block needing an images field for example would create it's own field once the migrate() method is called. That makes it possible to just drag&drop the folder of this block into another project and boom, everything is there. If I had a central place for migrations that would not be possible.

I know that this approach might look unconventional to some. So does ProcessWire itself with its "everything is a page" philosophy. But it works. Great.

On 11/27/2023 at 1:14 PM, d'Hinnisdaël said:

I'd prefer a module that limits itself to migrations, if not just for performance.

I invite anybody that actually used my module and notices a performance penalty to report it and I'll do my best to improve the module. I'm relying on RM every day (and have been for some years now), so reports like this would be highly appreciated. Thx.

BTW: There's also $config->useMagicPages = false ? 

[gebeer]

Thanks @bernhardfor the detailed explanations about splitting up the migrations. MAybe you want to add that to the Wiki. That would be awesome.

For that big project where I have that 4000 lines migrations in one file I still need to do that. When I started out with that project I wasn't fully aware on how to properly organize my migrations. Like with any other tool, knowledge grows with usage time ?

[bernhard]

Hey @gebeer it's really the same as with hooks. In the beginning placing all hooks in ready.php is the easiest. Later or on more complex projects it's better to move those hooks into their dedicated page classes.

The concept is simple:

• Everything that belongs logically to FooPage goes into FooPage::migrate()
• Everything that belongs logically to BarPage goes into BarPage::migrate()
• Everything that belongs to the project and nowhere else or that needs to take care of circular reference issues goes into Site.module.php

Site.module.php is a concept that I'm using for several years now and it is great. It's an autoload module that holds all the project specific stuff that belongs nowhere else. Similar to _functions.php but with all the benefits of OOP. Years ago I've created modules named after the project, like for the Foo project it was Foo.module.php and for the Bar project it was Bar.module.php; But I much more prefer Site.module.php which is why RockMigrations offers you to create this module for you (if you want).

Nowadays whenever I'm working on any of my projects I instantly now where to look for: Site.module.php - that saves brain power for more important tasks ? 

I've done a video about hooks and custom page classes and the same concept applies to migrations (the video starts at the interesting part):

All you have to do is make your custom pageclass a MagicPage and add a migrate() method:

<?php

namespace ProcessWire;

use RockMigrations\MagicPage;

class BasicPagePage extends Page
{
  use MagicPage;

  public function migrate()
  {
    $rm = $this->rockmigrations();
    $rm->migrate(...);
  }
}

Then as soon as you save that file those migrations will be fired. And only those. ? There's a reason why these features are built into the module...

[d'Hinnisdaël]

Appreciate the detailed explanation @bernhard and @gebeer, I was unaware of some of the finer points. Looking forward to trying the module again in the future.

A brief philosophical note among the technical discussion (the thread seems to have gone off track a bit ?) — RockMigrations is a great tool and people seem to be enjoying it. And not everybody has to agree with its way of doing things. We were merely expressing our views on what a good solution looks like. Choosing to use a particular tool is about preference and previous experience. It's about finding what works best for each of us, and that might change over time, or it might not. Neither is directed against you or a critique of your work. I have no doubt that the features I referred to as "a few things too many" are valuable in your workflow. That's the beauty about creating your own modules – you're free to tailor them to your needs and include whatever saves you time. My personal opinion is that these features shouldn't be part a migrations feature. You seem to disagree, and that's fine.

[Rasso]

Wow guys, that's a very lively discussion here! Appreciate it ?

@gebeer thanks so much for pointing out that support for RepeaterMatrix has landed in RockMigrations! I guess I must have just missed it in my last project. 

I'll certainly give RockMigrations another go after all the passionate feedback here ?  Since I know my forgetful self, a fully automated solution would be my gold standard:

• automatically generate and save migration files every time I create / update any field or template locally
• don't allow to edit or even access templates & fields in a production environment, at all

@bernhard do you have a pointer how I could setup that behavior using RockMigrations? I don't want to overstrain your patience, but if you have a few quick links for me I'd be very grateful. I'll make sure to RTFM ?

@MarkE I'll definitely also give your module a go when it's ready ❤️

[bernhard]

1 minute ago, Rasso said:

@bernhard do you have a pointer how I could setup that behavior using RockMigrations? I don't want to overstrain your patience, but if you have a few quick links for me I'd be very grateful. I'll make sure to RTFM ?

Hi @Rasso and also welcome to the forum. I'm always happy to answer questions and I always welcome constructive feedback. I just don't like if people post wrong information about my modules or put them in a wrong context.

Before I explain some things, please have a look at the intro video that I did one year ago. Some parts are outdated, for example the module will not automatically copy snippets to your project, it will ask you before doing so. But all the main concepts do still apply and will likely not change for a long time.

Actually @gebeer the video also shows how to move migrate() code into the pageclass. Totally forgot that ? 

 

25 minutes ago, Rasso said:

*automatically* generate and save migration files every time I create / update any field or template locally

That's not how RockMigrations works. At least not at the moment. And that has several good reasons. I'm not saying that such an approach is bad. But it is very complicated to implement, it comes with a lot of (maybe unsolvable) problems and even if everything worked it has a lot of limitations by design.

I didn't want that for RockMigrations, so I built it differently.

I can understand that it looks overwhelming to write migrations by hand. That's why RockMigrations does a few things more than only migrations. For example adding snippets (if you want and if you are using VSCode), so that you don't have to remember migration code by heart:

If anything is missing in those snippets you can just create that field manually and then inspect the field's migration code:

Or alternatively you will get the key-value pair by just hovering over the setting in 90% of the cases - for example changing the visibility state of the field to "open + locked (not editable)" you'd have to add "collapsed" => 6 to your migration code.

This might be a bit slower than using the gui. And it needs a little time to learn the basics. But once you get the concept it's often a lot quicker than working with the gui. Especially if you have a more professional setup and take the whole picture into account (having a staging / production system for example or working in a team).

But for a quick start: Just install the module and play around with /site/migrate.php

The beauty of that manual approach is that you only write code that you really need. Having a gui that stores a snapshot likely means that you get a bloated json or yaml with a trillion of settings that you don't need and that you might never understand, because you didn't take the time to inspect the whole json.

Manually writing migrations on the other hand means that you get beautiful diffs in your project's codebase:

If you can read diffs you can instantly understand what that migration does, right?

And if you have a look at line 71 you see that I'm using the Inputfield::collapsedHidden constant instead of the integer value 4 which you would get from a gui-based JSON/YAML migration file. I think that's much more readable and a lot easier to understand. And that's just another benefit of taking that extra step of writing migrations by hand instead of letting a tool do it for you.

Another huge drawback of a gui based migrations tool with a central place of migrations is that you limit yourself to the project. Everything you do you do for the project. With RockMigrations you can split migrations into reusable components and place migrations where they logically belong.

You might want to build a blog module that needs to create a "blog-overview" template and a "blog-post" template. Nice! Create those two pageclasses, add a migrate() method with all the template and field settings and voila, you have a module that you can reuse from project to project and only install it where you need it.

Imagine you have built that blog for your project with GUI based migrations... You have another project request with a blog? Have fun, do everything again. Of course that comparison is exaggerated, but you get the point.

RockMigrations is by far my most valuable module. It's stable, actively maintained for years and in daily use in many of my and many others' projects. I released it for free, because I think that this functionality should be built into the core and should be part of every more professional project. I thought we'd see more modules being released by the community, because RockMigrations makes it so simple to create modules that need fields/templates/etc. which is a pain to do with the API. I thought maybe Ryan might see what is possible and join the discussion. And I think it's a shame that ProcessWire is still said to be a good tool only for smaller projects.

Unfortunately I was wrong in many points. But fortunately that does not matter too much. RockMigrations is stable enough so that I don't need a core solution any more. I'm very happy with how it works and how well it plays together with ProcessWire and it's fundamental concepts.

On 11/27/2023 at 12:20 PM, Rasso said:

A clean solution for deploying fields between environments and committing them to version control.

RockMigrations offers exactly that.

If you try it and have any questions or you have any suggestions for improvements from your experience with WP/Kirby let me know.

[Rasso]

Woaa, thanks @bernhard for your extensive post. I guess with things like these, us devs tend to love what we know ?

I think your enthusiasm for RockMigrations is great, but I think it would be enough to describe the mechanics of your module without denigrating other approaches... otherwise you're doing the same thing you've accused others of in this post: criticizing a tool that you haven't used extensively yourself ? – it makes your post feel a bit like you are trying to sell something. Even though it's actually free and open source! ?

Quote

[...] you get a bloated json or yaml with a trillion of settings that you don't need and that you might never understand, because you didn't take the time to inspect the whole json.

• In WordPress+ACF, this is a non-problem. It's of course possible to create a plugin (the WP equivalent of a ProcessWire module) that contains definitions for Custom Post Types / Taxonomies (defined in code OR via GUI) as well as associated custom fields in JSON files (automatically generated) or even PHP files (the equivalent of your migrations files). The json/PHP files are not meant to be touched manually anyways, so no need to understand them – although they are easy enough to understand with some experience.
• In Kirby, it's absolutely different: there is no GUI for creating fields, anyways. All structure is being defined via code. So one could argue that their approach is not too different from using your module with writing migrations by hand, except they are using YAML wich was a total black box to me. Compared to that, your vscode snippets seem to be a clever solution.

Quote

Another huge drawback of a gui based migrations tool with a central place of migrations is that you limit yourself to the project. Everything you do you do for the project. With RockMigrations you can split migrations into reusable components and place migrations where they logically belong.

Not a problem in either WordPress or Kirby. Both allow to create re-usable templates, types and fields and put them in a plugin (=module)

Quote

If you can read diffs you can instantly understand what that migration does, right?

Yes I can read diffs ?. That's exactly how diffing would look in both WordPress and Kirby, as well. Only the things that changed, changed.

Anyhow... I feel, like after some time I'll learn to love whatever system I end up using for ProcessWire migrations. That's how us developers are built right? I'm definitely a victim of Stockholm Syndrome in that regard sometime ? 

 

[bernhard]

2 hours ago, Rasso said:

Not a problem in either WordPress or Kirby. Both allow to create re-usable templates, types and fields and put them in a plugin (=module)

Could you please explain or show how they do that?

[da²]

13 hours ago, Rasso said:

• automatically generate and save migration files every time I create / update any field or template locally
• don't allow to edit or even access templates & fields in a production environment, at all

I'm also interested in this use case, I'm also curious of the feasibility and I need a short "brain pause" from my current project, so I'm trying some experiments for fun.

Like @bernhard said this approach have limitations and drawbacks, at least:

• You must be sure that both databases use same ID for fields and templates (and maybe more like languages...),
  • But if that's not the case, we may fallback on field and template name, except when field/template has been renamed.
• Nobody can manually change fields and templates on the target installation,
• Some data can not be updated so easily, like the field type that can require changes on database structure,
• Maybe some data in "data" column ("fields" and "templates" database tables) can not be changed so easily too (I see for example some "parent_id" properties),
• Probably more drawbacks... ?

But anyway, I find the idea interesting and am having fun experimenting this.

Actually I have some bit of code that do 2 things, compare "fields" table from 2 PW installation and display a summary of the changes. Example of output:

Quote

[Fields]
────[Field 'qualificationResultFile2' (179)]
╌╌╌╌╌╌╌╌[name(UPDATED)=qualificationResultFile2, previous=qualificationResultFile]
╌╌╌╌╌╌╌╌[data]
╌╌╌╌╌╌╌╌╌╌╌╌[maxFiles(UPDATED)=2, previous=1]
╌╌╌╌╌╌╌╌╌╌╌╌[extensions(UPDATED)=json xml cvs, previous=json xml]
────[Field(ADDED) 'myNewField' (999), Array
        (
            [id] => 999
            [type] => FieldtypeFile
            [name] => myNewField
            [flags] => 0
            [label] => Fichier de résultat des qualifications
            [data] => Array
                (
                    [maxFiles] => 2
                    [extensions] => json xml
                    [outputFormat] => 0
                    [descriptionRows] => 0
                    [useTags] => 0
                    [collapsed] => 2
                    [defaultValuePage] => 0
                    [inputfieldClass] => InputfieldFile
                    [fileSchema] => 14
                    [description] => Le fichier de qualification permet de récupérer la position sur la grille (et calculer l'évolution, ACC et AC) ainsi que le temps de qualification (ACC, AC et rF2).
                )
        
        )
        ]
────[Field 'myAccountIRacingCustomerIdDescription' (183)]
╌╌╌╌╌╌╌╌[data]
╌╌╌╌╌╌╌╌╌╌╌╌[extraPlugins]
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌[3(ADDED)=addedPlugin]
────[Field 'raceResultsParsingSummary' (187)]
╌╌╌╌╌╌╌╌[data]
╌╌╌╌╌╌╌╌╌╌╌╌[newArray(ADDED)=Array
                (
                    [0] => test1
                    [1] => test2
                )
                ]
────[Field(DELETED) 'racePenalties' (180)]
────[Field 'raceResultsParsingSummary' (187)]
╌╌╌╌╌╌╌╌[data]
╌╌╌╌╌╌╌╌╌╌╌╌[extraPlugins]
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌[2(DELETED)=sourcedialog]
────[Field 'eventTop3' (191)]
╌╌╌╌╌╌╌╌[data]
╌╌╌╌╌╌╌╌╌╌╌╌[orderByCols(DELETED)=Array
                (
                    [0] => position
                )
                ]

Next step should be to update database, and adding sanity checks to avoid some kind of changes (like on field type).

But I'm not confident at all to use this on a production server before a lot of tests and a confirmation by Ryan and other PW developers that there's not risk updating what I'm gonna update. ?
I'm not even sure I'll use it one day, this is actually just experimentation for fun and curiosity.

[MarkE]

11 hours ago, bernhard said:

13 hours ago, Rasso said:

*automatically* generate and save migration files every time I create / update any field or template locally

That's not how RockMigrations works. At least not at the moment. And that has several good reasons. I'm not saying that such an approach is bad. But it is very complicated to implement, it comes with a lot of (maybe unsolvable) problems and even if everything worked it has a lot of limitations by design.

I didn't want that for RockMigrations, so I built it differently.

Quite correct, except "even if everything worked it has a lot of limitations by design". I'm not sure what those might be. ProcessDbMigrate is working pretty well now and I will let people know when I am happy with the testing (and documentation!). Yes, it was very complicated to implement but it is now coming together nicely. I think RockMigrations is great for people who want to work that way, but I chose a different path and will happily accept criticism (and, even better, constructive suggestions) if it doesn't work as intended.

11 hours ago, bernhard said:

Having a gui that stores a snapshot likely means that you get a bloated json or yaml with a trillion of settings that you don't need and that you might never understand, because you didn't take the time to inspect the whole json.

That is a risk, but I don't find it a problem with my module. It deliberately omits some unnecessary properties.

11 hours ago, bernhard said:

And if you have a look at line 71 you see that I'm using the Inputfield::collapsedHidden constant instead of the integer value 4 which you would get from a gui-based JSON/YAML migration file.

Yeah - that's nice. I wonder....

11 hours ago, bernhard said:

Another huge drawback of a gui based migrations tool with a central place of migrations is that you limit yourself to the project. Everything you do you do for the project. With RockMigrations you can split migrations into reusable components and place migrations where they logically belong.

ProcessDbMigrate uses multiple migrations and you can define one to export the database to another project. I am using it to install another module in new projects and plan to use  it to create a template project from a live one. I'll admit that it is early days yet and that it is probably not as flexible as RockMigrations, but I'm not convinced it is a 'huge drawback'.

11 hours ago, bernhard said:

Imagine you have built that blog for your project with GUI based migrations... You have another project request with a blog? Have fun, do everything again. Of course that comparison is exaggerated, but you get the point.

With ProcessDbMigrate, you use a Page class module in the same way (without all the field definitions etc.), create a very simple 'export' migration of the required fields and templates manually (just specify the names in repeaters), sync to your target and hit 'import'.

3 hours ago, Rasso said:

but I think it would be enough to describe the mechanics of your module without denigrating other approaches... otherwise you're doing the same thing you've accused others of in this post: criticizing a tool that you haven't used extensively yourself

?Agreed, but I don't blame @bernhardfor that. I would welcome his comments when my module is released.

3 hours ago, Rasso said:

The json/PHP files are not meant to be touched manually anyways, so no need to understand them – although they are easy enough to understand with some experience.

True in my case too, as the module interprets them and presents differences etc., but they have been useful in debugging ?

28 minutes ago, da² said:

Like @bernhard said this approach have limitations and drawbacks, at least:

• You must be sure that both databases use same ID for fields and templates (and maybe more like languages...),
• Nobody can manually change fields and templates on the target installation,
  • For both points above we may fallback on field and template name, except when field/template has been renamed.
• Some data can not be updated so easily, like the field type that can require changes on database structure,
• Maybe some data in "data" column ("fields" and "templates" database tables) can not be changed so easily too (I see for example some "parent_id" properties),
• Probably more drawbacks... ?

• ProcessDbMigrate uses names and paths not ids. If a name is changed it tracks that, so that it updates the target correctly. So ids do not need to be the same.
• You cannot change fields etc. in the target if they are the subject of a live migration (unless you disable that feature), but you can after it has been fully installed and 'locked'. Of course if someone updates the live then someone else creates a new migration from an old database version that over-writes it, that is a problem!
• So far I have not had a problem with any type of change that you can make in the UI (including parent id and template id references - these are converted to names and then converted back on installation)

Sorry if the above sounds a bit defensive, but I just wanted to set the record straight. Of course, you can disagree once the module is released and you have used it!

Just to emphasise a couple of points:

1. RockMigrations is a great free module and if it's what you want then use it.
2. ProcessDbMigrate does what I want and I thought it would be helpful to share it (but only when I am happy that it will not cause any difficulties). I have no desire ever to sell it or to position it as some sort of 'competitor', but I would be grateful if others do not knock it without just cause.

[Rasso]

Quote

Could you please explain or show how they do that?

• For WordPress+ACF it's pretty well explained here: https://www.advancedcustomfields.com/resources/local-json/#saving-explained. ProcessWire "Templates" do not really have a GUI in WordPress, they are being defined using various API functions like e.g. `regster_post_type`, `register_taxonomy`, ...
• With Kirby I have much less experience, but basically since everything is being defined using code, everything is in version control and synced between environments out of the box anyways. They have a very modular approach anyways, as you can see in their article about blueprints, where in the very first example it's already a plugin: https://getkirby.com/docs/reference/plugins/extensions/blueprints#file-definition

[da²]

@MarkE My approach is a bit different and very basic. I don't plan to migrate pages or anything else than fields and templates configuration, nor to interpret data or use PW API or track user changes with hook, it's just a raw database process. That's why there are more drawbacks and limitations.

If one day I need a real migration tool I'll use RockMigrations or ProcessDbMigrate. ?