New concept module: System Config Versions

[poljpocket]

Hi all. I have decided to publish one of the plugins I have been working on in my free time during the last few days but actually was motivated by our needs at work. It is currently in early beta and much of the functions might be subject to change. Before you jump at my throat and tell me that I am copying RockMigrations by @bernhard, please read on!

System Config Versions

This module adds an admin interface for developers to manage configuration revisions. Think of it as git for PW configurations ?.

It ensures that revisions only get run once. It is possible to run multiple revisions up to a certain point or even all available ones.

Aren't you just copying RockMigrations?

No. I made this plugin because at work, we needed more precise control over migrations and also do a lot more in migrations than adding&removing structural configurations. Often times, we use small snippets to change field data throughout the process of creating websites. And these may only be run exactly once.

Actually, we are usually using RockMigrations alongside this plugin to use it's very impressive function set! Think of this plugin as a way to persist run status for any migrations and ensure any migration only ever gets run once.

Simple Case Study

The motivation behind the module is best described with a code example:

<?php namespace ProcessWire;

/** @var RockMigrations $rm */
$rm = $modules->get('RockMigrations');

$rm->createField('richtext', [
    'type' => 'FieldtypeTinyMCE',
    // ...
]);

$rm->addFieldToTemplate('richtext', 'basic-page', 'textarea');

$pagesToUpdate = $pages->find('template=basic-page');
foreach ($pagesToUpdate as $pg) {
    $pg->setOutputFormatting(false);
    $pg->set('richtext', $pg->textarea);
    $pg->save();
}

$rm->removeFieldFromTemplate('textarea', 'basic-page');

The goal of this revision file is to convert an existing textarea field of all `basic-page`s to a richtext field. The code snippet which maps the contents of the fields is obviously part of the revision and with bare RockMigrations, we have no control over how many times it gets run. With SystemConfigVersions, the snippet gets run exactly once.

More Info & What's next?

For usage instructions and details, refer to the repo's README.

Do you guys have any first impressions or any recommendations or ideas for this project? Is there even a need for it in a broader sense?

You can find the GitHub repo here: https://github.com/poljpocket/SystemConfigVersions

Some ToDo's as of now:

• URGENT: Greatly improve error handling
• Add GitHub Actions support to automatically apply all new versions
• Add ability to reset run status down to a certain point
• Revise documentation below
• Add configurability for versions folder location and file naming scheme
• Add folder as version support (e.g. all files inside a version folder are executed and treated as one revision).

[elabx]

5 hours ago, poljpocket said:

Before you jump at my throat and tell me that I am copying RockMigrations 

Rather this looks a lot like Migrations module! Take a look you might be interested! Which btw I've been using for quite a few years along RockMigrations even if it shows it's deprecated, works just fine at least on PHP 7.4 which is where I'm stuck.

[MarkE]

Hi @poljpocket. It also has similarities to https://processwire.com/modules/process-db-migrate/ . That will run snippets before/after migrations. See the docs. 

[poljpocket]

Thanks you @elabx and @MarkE for your hints! I have looked at both these modules before we finally decided to go with RockMigrations, which for our workflow, works just as we need it. We like to create new templates, fields and their interconnections directly in the code and then let that flow from the local docker environment to the dev server and from there, on to the live site.

Sadly, I never considered Migrations to work alongside RockMigrations, I must give that one another try sometime. Also, since it explicitly states to be deprecated in favor of RockMigrations, we didn't look into it too much at all.

The reason we didn't go with DbMigrate was that although the feature set including even change-tracking are very strong and impressive features, it doesn't match our workflow at all. Mostly because we want to primarily work with files directly to build our structures even on the local environments.

All of that gave rise to the idea that we just needed this small little extension to somehow manage the revisions through either the admin or as a second step CI via GitHub.

But thanks to you, my biggest question is answered as there might not be any need on a broader spectrum since there are already some strong modules here. Of course I don't want other people to pick their poison between competing modules doing the same.

[bernhard]

Sounds cool! I understand the need for what you are working on and I'd even be happy to add such features to RockMigrations, if you want and think that could make sense, but more on that later ? 

22 hours ago, poljpocket said:

$rm->addFieldToTemplate('basic-page', 'richtext', 'textarea');

That was confusing for me because it's wrong ? Not trying to be know-it-all but it might be interesting for you to know that I tried to put all the method arguments in the order of how the method is named, so add-FIELD-to-TEMPLATE means field first, then template. Using PHP8 with named arguments that not that important any more, though.

So back to topic.

What you are trying to do have actually been my very first approaches when working with migrations. That's how they work everywhere after all. RockMigrations1 even had downgrade() and upgrade() methods and some version_compare to see if migrations should run or not. The reason why I completely removed that was because it has been so much more productive to use just config-based migrations in my daily work. Rather than writing all the logic around the actual migration to make sure migrations are only executed once I made sure that no matter how often they run they produce the same result.

That works great in 99% of my use cases, though there are situations where it does not and I admit it's not terribly beautiful ? So for example if you have fields added via RockMigrations that you don't need any more you'd do this:

<?php
public function migrate($rm) {
  // cleanup
  $rm->deleteField("old_fieldname");
  
  // actual migration
}

So while this would just work it's really not beautiful and it would log "Field old_fieldname not found" on every run. You can prevent that by adding TRUE as second parameter, but it's still not really beautiful.

I've also thought of adding something like this:

<?php
public function migrate($rm) {
  // cleanup
  $rm->once(function($rm) {
    $rm->deleteField('old_fieldname');
  }
}

The problem here is: How does RM know, what that "once" means? What if it fails? What if inside the once was multiple steps like in your example, for example looping over 100 pages and after 50 it breaks? What do you in your module in such a case?

What I've done in such cases was to create an import script based on RockShell that I could run whenever needed. Manually. In the best case only once ? I could also run this script locally as often as I want, try everything out, see the result, and if it's not working I'd just do "rockshell db:restore" and try it again. Once everything works I push to production and run the script there.

This is what your script would look like in RockShell:

<?php

namespace RockShell;

use function ProcessWire\rockmigrations;

class DemoMigration extends Command
{

  public function handle()
  {
    $rm = rockmigrations();

    $rm->createField('richtext', [
      'type' => 'FieldtypeTinyMCE',
      // ...
    ]);

    $rm->addFieldToTemplate('richtext', 'basic-page', 'textarea');

    $pagesToUpdate = $this->wire()->pages->find('template=basic-page');
    foreach ($pagesToUpdate as $pg) {
      $this->write("Updating page #$pg ...");
      $pg->setOutputFormatting(false);
      $pg->set('richtext', $pg->textarea);
      $pg->save();
    }

    $rm->removeFieldFromTemplate('textarea', 'basic-page');
    $this->success("Hooray - we are done :)");

    return self::SUCCESS;
  }
}

Put that in /site/assets/RockShell/Commands and you are good to go.

I'd also really be interested in how you work with docker. I've seen https://github.com/poljpocket/processwire-docker and your statement 

13 hours ago, poljpocket said:

and then let that flow from the local docker environment to the dev server and from there, on to the live site.

sounds very interesting. So I'd love to hear more about that if you find time.

[poljpocket]

Thanks Bernhard for your post!

55 minutes ago, bernhard said:

That was confusing for me because it's wrong ?

I just typed that out here, it isn't something I copied from somewhere. And it must have been late after a long day at work ? Of course this isn't right and I have fixed it above!

I love separation of concerns. I view your RockMigrations as an API wrapper which lets me control templates and fields in a very cool way and you have support for RepeaterMatrix which is something we use quite often. Also because of this, I am - just a tad - bothered that you included MagicPages and some "random" QOL features in RockMigrations which cover bunch of completely different concerns. I would offer MagicPages as an optional extension to RM and move all the QOL stuff into a new module. But at the end, I am glad, you put in all the work for RM, so I'll stop bashing you now ?! RM is tailored to your workflow which is completely fine.

In order for you to understand where all these ideas come from, I need to explain our normal project flow first:

Usually, one person in my team starts the project and will setup a local installation of PW using this repo (you pointed that out already) and will use a site profile like this one. We can move this repo around in our team and work on it collaboratively. The setup will use a small set of "test data" which rarely changes or reflects the data which will comprise the end result. These will largely be placeholder media and lorem ipsum texts.

Sometime during the early to mid-maturity of the project, our content people want/need to start working on the contents and our designers will want to insert all the images and media. The larger the project, the earlier this will happen. This is the point where we start to have two installations: one local and one on our dev server. From this point on, we cannot touch the pages and their contents anymore and right here, migrations are the way we need to "upgrade" the dev server to keep it in line with our local version. This is also why we need these small snippets which can only be executed exactly once.

Since it is rarely the case that at this point, the project is feature-complete, my example above is quite usual: We have the template use a textarea field but due to a late spec change, we have to change this to a TinyMCE field somewhere down the road. Since there could be many pages holding final content already, we need to use the snippet to move the data around. Also, for debugging issues on the dev server, we might copy some of the final content down to the local environment by hand to have some more context.

From there, in the very late stages of the project, the site goes live and might need some more touch-ups before the launch. This is usually the point where a migration might make it's way from docker to the dev server and then to the live site.

As you can see, we might have three teams wanting to push the project forward at the same time. This calls for exactly what my module is about.

[poljpocket]

By the way, I had a private conversation with @elabx in which we discussed the Migrations module a bit more. Since it said "deprecated in favor of RockMigrations", I originally didn't look into it further. This was a mistake! It basically does exactly what my module does. Just better. And there is absolutely no reason not to combine it with RockMigrations!

I have forked that module and put in some early work to update it to namespaces and PHP7/8 compatibility. I might talk to @LostKobrakai to maybe take over or provide some PRs to keep it alive since there is a commercial incentive now.

https://github.com/poljpocket/Migrations/tree/pw3-php8 note the work is just on a separate branch for now.

[bernhard]

1 hour ago, poljpocket said:

In order for you to understand where all these ideas come from, I need to explain our normal project flow first:

Thx for trying to explain the details. I understand you well. This is why I built RockMigrations.

1 hour ago, poljpocket said:

From this point on, we cannot touch the pages and their contents anymore and right here, migrations are the way we need to "upgrade" the dev server to keep it in line with our local version. This is also why we need these small snippets which can only be executed exactly once.

On the last sentence I'm losing you. I'd still be interested in a detailed example of what you are doing and where this is necessary and why. You are explaining extensively the same workflows that I'm using and then you suddenly state that something is missing without going into detail.

For example: How to you execute these snippets? Who does it? Did you think of any different approaches? Why do you think this has not been an issue for me so far?

[poljpocket]

Hi Bernhard, lots of questions ? I am happy to answer!

44 minutes ago, bernhard said:

How to you execute these snippets?

Right now, we are using my module and in the not so distant future, probably the Migrations module.

44 minutes ago, bernhard said:

Who does it?

Always me, the team lead. We are planning to use GitHub actions soon though.

46 minutes ago, bernhard said:

Did you think of any different approaches? Why do you think this has not been an issue for me so far?

I think this workflow makes sense and I can't come up with a different approach. Sure, most of the migrations probably work just fine if executed several times and for those, we could use vanilla RockMigrations. I can even give you two good examples. Note that I can't share code since these are both commercial projects:

Featured images
Right now, we have an ongoing project where we as a first phase, finished the editing experience for a products list (not a shop, but many of the same features). We have included a product gallery and for the longest time, it was fine to just use the first image uploaded as the featured image. Then, we ran into some problems with images having unfortunate subject placements which made the first image with the same settings not compatible with the featured image concept as it was. This forced us to create a new image field where we can set another focus point and sizing constraints. Since the client had finished inserting 200+ products already, we simply cannot ask them to just upload the first image (or possibly a new one) for every product again. The snippet in question which only can be ran exactly once is:

1. "duplicate" the first image on disk,
2. resize to match new constraints,
3. set it as the featured_image,
4. copy over focus point settings (which is fine for most, but not all).

Everything else (create field, add to template, ...) is done with RockMigrations.

Why can I run this only exactly once: From this point on, the client will start uploading possibly different images to featured_image and change some of the settings where the original ones don't work out. If we ran this procedure a second time, all that work is gone and the filesystem/database is possibly in a corrupted state.

Combining multiple fields into one with specific markup
For another project, we had some references set up with a very sophisticated mask for inserting info like the client, the year and some more meta information. Some spec change for this forced us to combine all of this information into a single rich editor to allow for more flexibility and customization. Again, much of the content management has already been done at this point and we couldn't just ask to do it again the "new way". The snippet which only can be ran exactly once is:

1. render the fields together as markup and set to the body field
2. remove all fields not needed anymore (with RockMigrations)

Again, everything else (add to template, configure field in context, ...) is done with RockMigrations.

Why can I run this only exactly once: This one is a bit less clear as this migration might be gated with a simple if (e.g. are the fields still on the template or not). But still, it is easier to just make sure it doesn't run more than once.

[bernhard]

Thx. I'd just created a RockShell script in both cases and ran that on the server. So it looks like you are rebuilding RockShell, not RockMigrations ? But yeah, if you want a GUI, go for it ? 

[poljpocket]

True that. We have yet to take a look at RockShell. But in any case, the reason for the GUI is simple and necessary: shared hostings

Of course this is only true as long as we don't use Webhooks.

[bernhard]

In that case it makes sense, I agree. Now we finally identified the important differentiation ? 

I could also think of a GUI for RockShell that makes it possible to run commands directly from within the PW backend. I could even think of it pushing live feedback via SSE to a console in the backend. And there could also be helpers to run commands from Github actions.

That would be a great way to contribute and give something back to something that you get for free.

And you should really have a look at the "db:restore" and "db:pull" commands. They are gold when working with migrations and not only then. For example I've so often had the situation where I quickly wanted to play around with something. For example install a new module or show something to a friend/colleague on a meeting.

With RockShell you can do all that without worrying. Add fields, install modules, etc etc. When you are done you simply do "rockshell db:pull" and you have the exact same state as before. Oh, and what about the files that the module installed? Just revert changes of your git project and that's it.

[elabx]

5 hours ago, bernhard said:

I'd also really be interested in how you work with docker. I've seen https://github.com/poljpocket/processwire-docker and your statement 

19 hours ago, poljpocket said:

Sorry for the offtopic, but will second vote for this ? Do you happen to use docker in production?

[poljpocket]

No not at all. Just for local development. EDIT: not at all = no, we don't use docker for deployments.

Main reason: Most of our clients won't pay for the increased cost of a cloud platform.

Second important reason: Most of our projects in the end are "just websites" and for that, the bells and whistles a shared hosting environment provides on the cheap are just too good to not use (e.g. email sending without a complicated configuration, managed configuration and automatic updates, and HTTPS support with auto certificate renewal).

Also, you can look at some of the other docker topics I have answered on the forum, PW doesn't play too well with cloud environments (e.g. problems with session retention due to the proxying CEs use). I could be wrong, but I haven't digged into it more than I needed to in order to have a clean and simple local docker environment.

Edited February 7 by poljpocket
read question wrong at first

[bernhard]

On 2/7/2024 at 10:20 AM, bernhard said:

I've also thought of adding something like this:

<?php
public function migrate($rm) {
  // cleanup
  $rm->once(function($rm) {
    $rm->deleteField('old_fieldname');
  }
}

The problem here is: How does RM know, what that "once" means? What if it fails? What if inside the once was multiple steps like in your example, for example looping over 100 pages and after 50 it breaks?

I've gone ahead and added this to the DEV branch! Please see