RockMigrations1 - Easy migrations from dev/staging to live server

[bernhard]

This is the thread for the old version of RockMigrations wich is deprecated now and has been renamed to RockMigrations1

Here is the latest version of RockMigrations:

-- Old post from 2019 --

https://github.com/baumrock/RockMigrations1

Quickstart

• First make sure you have backups of your database!!
• Install RockMigrations
• Create this ready.php

<?php
/** @var RockMigrations1 $rm */
$rm = $this->wire('modules')->get('RockMigrations1');
$rm->migrate([
  'fields' => [
    'myfield' => ['type' => 'text'],
  ],
  'templates' => [
    'mytemplate' => [],
  ],
]);

• Open your PW Backend and you'll see the new field and new template!
• Now add the new field to the template:

<?php
/** @var RockMigrations1 $rm */
$rm = $this->wire('modules')->get('RockMigrations1');
$rm->migrate([
  'fields' => [
    'myfield' => ['type' => 'text'],
  ],
  'templates' => [
    'mytemplate' => [
      'fields' => [
        'title',
        'myfield',
      ],
    ],
  ],
]);

• Reload your backend and inspect the template:

• Now let's add a label for our new field and make title+myfield 50% width:

<?php
/** @var RockMigrations1 $rm */
$rm = $this->wire('modules')->get('RockMigrations1');
$rm->migrate([
  'fields' => [
    'myfield' => [
      'type' => 'text',
      'label' => 'My Field Label',
    ],
  ],
  'templates' => [
    'mytemplate' => [
      'fields' => [
        'title' => ['columnWidth' => 50],
        'myfield' => ['columnWidth' => 50],
      ],
    ],
  ],
]);

• That's all the magic!

You can easily lookup all necessary properties in Tracy Debugger:

Let us add a Textformatter as an example! Add the textformatter via the PW backend (not via RM), save the field and inspect via Tracy:

Now add this data to your migration:

<?php
/** @var RockMigrations1 $rm */
$rm = $this->wire('modules')->get('RockMigrations1');
$rm->migrate([
  'fields' => [
    'myfield' => [
      'type' => 'text',
      'label' => 'My Field Label',
      'textformatters' => [
        'TextformatterEntities',
      ],
    ],
  ],
  'templates' => [
    'mytemplate' => [
      'fields' => [
        'title' => ['columnWidth' => 50],
        'myfield' => ['columnWidth' => 50],
      ],
    ],
  ],
]);

Ok maybe you noticed that migrations are now running on every request which may slow down your site a lot! RM2 will have a feature to automatically detect changes and fire migrations automatically. RM1 does not have this feature, but I've used a technique in all my projects that fires migrations on every modules refresh. Simply wrap the migration in a fireOnRefresh method call:

<?php
/** @var RockMigrations1 $rm */
$rm = $this->wire('modules')->get('RockMigrations1');
$rm->fireOnRefresh(function() use($rm) {
  $rm->migrate([
    'fields' => [
      'myfield' => [
        'type' => 'text',
        'label' => 'My Field Label',
        'textformatters' => [
          'TextformatterEntities',
        ],
      ],
    ],
    'templates' => [
      'mytemplate' => [
        'fields' => [
          'title' => ['columnWidth' => 50],
          'myfield' => ['columnWidth' => 50],
        ],
      ],
    ],
  ]);
});

How to remove things you created before you may ask? Use the declarative syntax:

<?php
/** @var RockMigrations1 $rm */
$rm = $this->wire('modules')->get('RockMigrations1');
$rm->fireOnRefresh(function() use($rm) {
  $rm->deleteField('myfield');
  $rm->deleteTemplate('mytemplate');
});

Refresh your backend and everything is as it was before!

PS: Make sure you have proper Intellisense support in your IDE to get instant help:

[kongondo]

2 hours ago, bernhard said:

Sorry, I'm in a hurry and have to leave, but I'm happy to hear your thoughts! ? 

I like the productivity of this lazy man who is in a hurry ?!  You are on fire mate! ?.

[LostKobrakai]

19 hours ago, bernhard said:

The second point does bring downsides with it, so this might have been intended by him.

Quite correct ? Migrations are supposed to be immutable files, so running them always yields the same effects. This is best accomplished if those migrations are managed in one central place by the entity affected. Nothing outside should implicitly be able to affect what migrations do. Having migrations in modules however does exactly that. If you update a module the migrations in it might be updated, which is not good (it might have already been executed in prod with the old code and later migrations depending on the new code might fail). Even having immutable migrations in the module might quickly become a burden as each new user must be moved through all „mistakes“ and „changes“ as well before being migrated to a final state needed by the most current version.

While I‘ve talked a bit about the mentioned downsides I also think modules should come with migrations, but in a different form. There should be generators, which generate migrations in the central migrations folder of the site using the module. They imho should always just generate the most recent version of migration needed. If a module does change the way to update old clients should be by having migrations to update from e.g. V1->V2. Those could be put in another generator or even just be put into a changelog. New users of a module wouldn‘t need to care about it, they directly get migrations generated for V2, existing users can leave their existing migration for V1 in their system and just add the update migration for V1->V2.

This way one can have the benefits of modules supplying migrations to users, while not having the downsides and a imo simpler option for not needing to keep track of all the old stuff already changed till eternity, especially for new users, which really don‘t need to care about old versions.

One last benefit of generation of code is that it‘s editable by the user. Like I could add localized labels to a field generated for a certain module, without any extra effort needed by the module creator. Another example might be adding additional fields to a template a module uses/creates, which might be needed just on the single site. This is also a downside, as editing could break the functionality, but I think it’s a quite clear danger and one can always generate the file(s) again to get to a working migration. 

Edit:
One last point. Migrations in modules like they're intended by @bernhard can work as well. I hope it's clear that most issues I pointed out above are about control, the one for the actual system using the migrations. I'm not keen on automatic stuff happening, which could break my system. Others might be happy to let module creators be in charge of not skrewing things up. On the other hand control means more responsibility, like changes in versions are a little bit more work for the module user.

[bernhard]

Thx for clarifying.

I don't like the idea of code generators. Maybe because I'm not familiar with them. But I could think of some kind of central migrations queue system. Modules could "register" their migrations in a central place and make sure the order of execution is in a way that there are no conflicts. For example a migration could define

requires => ['otherModule' => '0.0.3']

This could then run the migration in "myModule", execute "otherModule" until v0.0.3 and continue with executing "myModule" migrations afterwards. At the moment this is a little theoretical, because I usually do small iterations, push them to production and for the next iteration I have to identical states to start from. That's easy to handle. Sometimes I even do changes by hand, if that is no problem (eg the change does not affect the live system - like changing a typo in some textfield or the like).

Also running a migration twice is usually not a problem at all. My api is designed in a way that, for example, if you create a field in your migration and the field already exists, it will just return that field. I'm not sure about this one. At the moment it feels like this is not the best idea, but I'm sure I've had a reason to do it this way ? It's alpha... things might change. I released it to get some feedback and input. And so far it did a great job on several updates across several sites ? 

[LostKobrakai]

23 hours ago, bernhard said:

I don't like the idea of code generators. Maybe because I'm not familiar with them.

My migrations module has generators. Basically everything creating a migration file for you is one. Just that those are meant to be edited further, while files generated by modules would work as is.

Generally I think you‘re quite fine as you‘re doing modules and the sites using them by yourself. It‘s getting more complicated if things are no longer in one hand and happen independently from each other.

[bernhard]

Not sure if anybody is already using it... I'm using it every day and I'm extending the api from time to time. Today I added a setFieldOrder() method. Now it is quite easy to change the order of fields and set new column widths:

// adjust field widths
$rm->setFieldData('project', ['columnWidth'=>33], 'rockinvoice');
$rm->setFieldData('inv_date', ['columnWidth'=>33], 'rockinvoice');
$rm->setFieldData('type', ['columnWidth'=>34], 'rockinvoice');

$rm->setFieldData('invoicetomarkup', ['columnWidth'=>100], 'rockinvoice');
$rm->setFieldData('due', ['columnWidth'=>25], 'rockinvoice');
$rm->setFieldData('paid', ['columnWidth'=>25], 'rockinvoice');
$rm->setFieldData('num', ['columnWidth'=>25], 'rockinvoice');
$rm->setFieldData('sap', ['columnWidth'=>25], 'rockinvoice');

$rm->setFieldOrder([
  'type',
  'invoicetomarkup',
  'due',
  'paid',
  'num',
  'sap',
], 'rockinvoice');

 

[bernhard]

Just added support for Fieldsets. They work a little differently than normal fields, because they need a corresponding _END field. Now if you create fieldsets in your migration those fields will automatically be created and automatically be added to a template:

Upgrade:

// create tab and add it to invoice template
  $f = $rm->createField('dunningtab', 'FieldtypeFieldsetTabOpen', [
    'label' => 'Mahnwesen',
  ]);
  $rm->addFieldToTemplate($f, 'invoice');
  
// create sample field and add it to this tab
  $f = $rm->createField('invoiceinfo', 'FieldtypeRockMarkup2', [
    'label' => 'Info zur Rechnung',
  ]);
  // add it after the field "dunningtab", before the field "dunningtab_END"
  $rm->addFieldToTemplate($f, 'invoice', 'dunningtab');

Downgrade:

$rm->deleteField('dunningtab');
$rm->deleteField('invoiceinfo');

This will remove your field from all templates where it is used and also remove the corresponding closing field ? 

[bernhard]

Added support for renaming templates (and related fieldgroups): https://github.com/BernhardBaumrock/RockMigrations/commit/fd465dc27e4f2e3982ed6669da0478df8addfd90

$newTemplate = $rm->renameTemplate('oldtemplate', 'newtemplate');

 

[bernhard]

It has been quiet here, but I'm using RockMigrations on a daily basis. It can also be handy during site development, eg today I had to refactor my setup (change some fields, delete them, create new fields instead). I only have test data on this site, so when I want to delete a field, I really don't care about losing data...

PW asks a lot of questions before removing a field from the system (which is good in 90% of the cases), but for me it is often a burden. RockMigrations + TracyDebugger to the rescue:

$rm = $modules->get('RockMigrations');
$rm->deleteField('location');

Manually one would have to remove the field from the templates first, confirm that one might lose data etc.; Too much clicking imho ? 

[adrian]

16 hours ago, bernhard said:

Manually one would have to remove the field from the templates first, confirm that one might lose data etc.; Too much clicking imho

Or in this case you could just use Tracy's Admin Tools panel which has a one-click "delete field" button when viewing a field's setting in the admin ?

PS - not meaning to be a downer on your module in general, it's just not necessary for this use case. 

[bernhard]

Nice! Didn't know that! Always happy to hear about other options that I might not know, thx! ?

[dragan]

17 hours ago, adrian said:

Or in this case you could just use Tracy's Admin Tools panel which has a one-click "delete field" button when viewing a field's setting in the admin

OMG... wish I knew that some such shortcut existed before.

[adrian]

1 minute ago, dragan said:

OMG... wish I knew that some such shortcut existed before.

We should end our OT ishness ? but there are other similar shortcuts in that panel as well - worth keeping open and seeing what it offers up in various parts of the PW admin. Let me know if you have any other suggestions for features for it.

[aComAdi]

We use RockMigrations regularly for adding custom functionality, pages, templates and fields to new projects and update the existing sites modules. Works a treat. 

[bernhard]

Thx for the report - always nice to hear that it does not only work for me ? 

You should definitely take a look at the quite new migrate() method that lets you create fields, templates and pages via simple array syntax ? I'm using this in every module now and it saves me so much time ? 

/**
* Setup this module
*/
public function setup() {
  $this->rm->migrate([
    'fields' => [
      'field_done' => [
        'type' => 'textarea',
      ],
    ],
    'templates' => [
      'template_triggers' => [
        'fields' => ['title'],
        'icon' => 'database',
        'noParents' => -1, // only one
        'childTemplates' => ['template_trigger'],
        'noChildren' => 1, // we don't allow pages to be created via backend
        'sortfield' => '-created',
        'tags' => 'RockTrigger',
      ],
      'template_trigger' => [
        'fields' => [
          'title',
          'field_done',
        ],
        'icon' => 'bolt',
        'noChildren' => 1,
        'parentTemplates' => ['template_triggers'],
        'pageClass' => '\RockTrigger\TriggerPage',
        'noSettings' => 1,
        'tags' => 'RockTrigger',
      ],
    ],
    'pages' => [
      'rocktriggers' => [
        'title' => "Triggers",
        'template' => 'template_triggers',
        'parent' => 1,
        'status' => ['hidden', 'locked'],
      ],
    ],
  ]);
}

PS: In my project this setup() method is triggerd on every modules refresh. Maybe a little overhead but for now the easiest option...

[bernhard]

Just pulled a PR from @Craig, updated the readme, bumped version to 0.0.8, removed the beta flag and submitted the module to the modules directory ? 

I'm using this module now on all my projects and I'm happy to get some feedback if anybody else is using it. Stars/Donations welcome: https://github.com/BernhardBaumrock/RockMigrations/stargazers THX

[LostKobrakai]

I'm actually wondering if I should deprecate my migrations module in favor of yours. I'm hardly using processwire professionally anymore and while my migrations module is stable and will likely continue to work I won't do any considerable updates anymore. To me it feels your module does everything my module does and a bunch of things more besides the fact you're still actively invested in it. Do you think that's a fair statement?

[bernhard]

Personally I don't care, because I'm using my version only (of course). But for someone new to migrations and having to decide between the two options your points might be a good reason to choose RockMigrations. So for them it might be good to know, yes ? 

[LuisM]

1 hour ago, LostKobrakai said:

I'm actually wondering if I should deprecate my migrations module in favor of yours. I'm hardly using processwire professionally anymore and while my migrations module is stable and will likely continue to work I won't do any considerable updates anymore. To me it feels your module does everything my module does and a bunch of things more besides the fact you're still actively invested in it. Do you think that's a fair statement?

@LostKobrakai @bernhard sorry for the Off-Topic here, as I know both of your modules and follow both of your ProcessWire work for quite some time now.

Would you mind to elaborate a bit on why you are not using PW anymore and what you have choosen instead? 

[bernhard]

1 minute ago, LuisM said:

@LostKobrakai @bernhard sorry for the Off-Topic here, as I know both of your modules and follow both of your ProcessWire work for quite some time now.

Would be great to hear your experience then. Is @LostKobrakai's module really obsolete or does it still have its place?

[LuisM]

4 minutes ago, bernhard said:

Would be great to hear your experience then. Is @LostKobrakai's module really obsolete or does it still have its place?

Well to quote Mr. Wilde 

Quote

To me it feels your module does everything my module does and a bunch of things more

I would second this.

On my Job I had to venture in the Lumen/Laravel ecosystem a bit and really liked the Migrations approach they take. (https://laravel.com/docs/7.x/migrations) 
I really missed a proper migration integration when I started some new PW projects in the past. 

@LostKobrakai's approach was therefor my way to go. But ProcessWire is evolving and the lack of support he could give is a strong driver to switch. 

[LostKobrakai]

4 hours ago, LuisM said:

I would second this.

That's good to hear. I've never actually tried out RockMigrations, so I wasn't really sure if functionality matches up like I expected it to do.

4 hours ago, LuisM said:

Would you mind to elaborate a bit on why you are not using PW anymore and what you have choosen instead?

Quite simple: I no longer feel it's a great fit for the stuff I'm developing, which is custom web applications opposed to websites. I've not only switched framework, but also language to using elixir and the phoenix framework for most of my work. I still use processwire for some websites, but they change rarely so they're not really the ones invoking changes to my modules.

[Pixrael]

7 hours ago, bernhard said:

Stars/Donations welcome

The Github Sponsors program is still in beta. We need to put a link for donations in each profile of the Developer Directory.

[zoeck]

Very nice module! I've been using it for a few days now and I'm getting more and more excited every day.

Once you know how to use everything, working with it is really fun, especially when you create an update for a larger project.

[zoeck]

Are there any problems when you create repeaters and use fields that are created in the same upgrade?

There is no error, but my fields were not assigned to the repeater.

 

Here's my Code... 

The Fields ""repeaterFields" => ["proj_milestone_type", "proj_milestone_date", "proj_milestone_done", "proj_milestone_text"]" are created before the repeater, but in the "same field creation step".

            'proj_milestones' => [
                'type' => "Repeater",
                'label' => 'Meilensteine',
                'tags' => "Projekte",
                'repeaterCollapse' => 3,
                'repeaterLoading' => 1,
                "repeaterFields" => ["proj_milestone_type", "proj_milestone_date", "proj_milestone_done", "proj_milestone_text"],
                "repeaterMaxItems" => 16,
                "repeaterTitle" => "{proj_milestone_date} {proj_milestone_type}",
                "repeaterAddLabel" => "Meilenstein hinzufügen",
                "icon" => "calendar",
                "fieldContexts" =>  [
                    "proj_milestone_type" =>  [
                        "columnWidth" =>  20
                    ],
                    "proj_milestone_date" =>  [
                        "columnWidth" =>  20
                    ],
                    "proj_milestone_done" =>  [
                        "columnWidth" =>  20
                    ],
                    "proj_milestone_text" =>  [
                        "columnWidth" =>  40
                    ]
                ]
            ],

 

 

And a second problem with an option field:

'proj_milestone_type' => [
                'type' => "Options",
                'label' => 'Meilenstein',
                'tags' => "Projekte",
                'inputfieldClass' => "InputfieldSelect",
                'export_options' => ['default' => '1=Bestelleingang\n2=Kickoff\n3=Konstruktion Mechanik'],
            ],

The options were not applied to the field ?