RockMigrations1 - Easy migrations from dev/staging to live server

[bernhard]

Well... I've read your post 3 times and think I understand what you mean.

I don't think that there is any common definition what a migration exactly is. There are too many different scenarios where the term can be used: https://en.wikipedia.org/wiki/Migration

I'm mainly using it in my modules nowadays with the migrate() function to get consistent field/template/page setups that I can use and quickly change in the codebase that is version controlled:

$this->rm()->migrate([
  'fields' => [
    'foo' => [
      'type' => 'textarea',
      'label' => 'Text1',
      "inputfieldClass" => "InputfieldCKEditor",
      "contentType" => 1,
    ],
  ],
]);

This could simply be changed and committed:

$this->rm()->migrate([
  'fields' => [
    'foo' => [
      'type' => 'textarea',
      'label' => 'Text1',
      "inputfieldClass" => "InputfieldCKEditor",
      "contentType" => 1,
    ],
    'bar' => [
      'type' => 'textarea',
      'label' => 'Text2',
      "inputfieldClass" => "InputfieldCKEditor",
      "contentType" => 1,
    ],
  ],
]);

Then simply run a git pull on the server and I have the same setup as locally.

[benbyf]

I guess in that case its solving the big content VS config divided present in all CMS's. But yes, i am cyrptic it seems. I guess my point was if I dont need the auto upgrading and downgrading that the module provides then I could not use it and simply write some template and field php code that you can cool as a "migration" as your module is a wrapper for quickly doing those tasks (exlcuding the uping and downing).

In an ideal world the changes would be recorded in situ (no need to write them) and migrations would be exported to file to added to version control that can be ingested by any other PW install running the module. (I'm just all about writing less)

[bernhard]

9 minutes ago, benbyf said:

In an ideal world the changes would be recorded in situ (no need to write them)

I think everybody who has ever used the macro recorder of msoffice/vba knows that this world is not ideal ? But I think I know what you are talking about - I've been there ? 

 

[benbyf]

HA! you do know what i mean then!!! ? well I'll shut up then.

[Sascha Nos]

On 7/22/2020 at 11:54 PM, sebr said:

I created a module and I use the auto-update system from RockMigrations (0.0.1.php). All is OK with the install and the uninstall. I create a 0.0.2.php and update the version of the module ('version' => '0.0.2'). But when I refresh modules from Processwire, my module update in 0.0.2, but no RockMigrations is executed.

Hey there, experiencing the same issue. Is there any solution?

Thank you so much! Sascha

[Sascha Nos]

The migration gets only executed when the module is loaded. You need to add

'autoload' => true,

to your module.

[Richard Jedlička]

Hi @bernhard, thanks for a great module. I like the migrate function for declarative definition of PW setup.

1) I saw in your example you are using ModuleName.config.php files to store the config for migration. I also saw the in other PW examples the file with the same name could be used for ConfigurableModule type of module, but as I understand it has different meaning. I know I can name the file as I want but Is it a good practice to use this file for migration (and show it in examples)? 

2) Could you please add ability to set different name of directory with migrations? I'd like to use just migrations instead of RockMigrations. 

3) It would be great for the module to be easily installable by composer, it is quite easy, see https://processwire.com/blog/posts/composer-google-calendars-and-processwire/#composer-for-module-developers

 

[bernhard]

Hi @Richard Jedlička, thx for your message ? 

2 hours ago, Richard Jedlička said:

1) I saw in your example you are using ModuleName.config.php files to store the config for migration. I also saw the in other PW examples the file with the same name could be used for ConfigurableModule type of module, but as I understand it has different meaning. I know I can name the file as I want but Is it a good practice to use this file for migration (and show it in examples)? 

Thx, that's a good point! I've added a new example file of how I am using RockMigrations nowadays that should be really helpful to get started with the power of RockMigrations and building easy, fast and reusable modules: https://github.com/BernhardBaumrock/RockMigrations/blob/master/examples/MigrationsExample.module.php

2 hours ago, Richard Jedlička said:

2) Could you please add ability to set different name of directory with migrations? I'd like to use just migrations instead of RockMigrations. 

You might not need this feature any more once you saw the new example file, but anyhow it was an easy update and I've made the getMigrationsPath() method hookable so you can use whichever folder you like ? 

2 hours ago, Richard Jedlička said:

3) It would be great for the module to be easily installable by composer, it is quite easy, see https://processwire.com/blog/posts/composer-google-calendars-and-processwire/#composer-for-module-developers

I'd be happy to get a quick tutorial of how all that works and what the benefits would be and then I can take that into account. PRs are always welcome.

[Richard Jedlička]

Thank you for quick reply! I will take a look at it.

1 hour ago, bernhard said:

I'd be happy to get a quick tutorial of how all that works and what the benefits would be and then I can take that into account. PRs are always welcome.

I see composer installation really useful because I've just specified all dependencies in composer.json file and don't have to commit modules into my site's source code. So, when I deploy the site to production the automated script easily install all dependecies. All I have to do manually is to install the modules via admin (maybe this could be automated as well) or refresh if they are already installed. I've even created a "wrapper" composer package for processwire (see https://github.com/uiii/processwire), which will install like a regular installation, or upgrade the existing. So my site's source code only contains the site folder (without modules) and composer.json (and composer.lock) and this is enough basis to setup the whole running site by single command:

composer install

 

[bernhard]

10 hours ago, Richard Jedlička said:

All I have to do manually is to install the modules via admin (maybe this could be automated as well)

This is really easy either via RM or via default API:

$modules->get('TracyDebugger');
// or
$rm->installModule('TracyDebugger');

10 hours ago, Richard Jedlička said:

I see composer installation really useful because I've just specified all dependencies in composer.json file and don't have to commit modules into my site's source code.

I'm using a GIT setup with submodules. So all I have to do is

git submodule update --init --recursive

That way I always have my modules under control of GIT and can efficiently commit new changes from any project that I'm currently working on ? 

But I went ahead and added support for rockmigrations via composer: https://packagist.org/packages/baumrock/rockmigrations

Would be interested to hear if everything works!

[Richard Jedlička]

8 minutes ago, bernhard said:

This is really easy either via RM or via default API:

Ah, good!

8 minutes ago, bernhard said:

I'm using a GIT setup with submodules. So all I have to do is

Yes, this is also the way. But I see composer as de-facto standard for maintaining dependencies, not only PW modules. I'm used to npm in JS world and love it, so composer is the way for me.

10 minutes ago, bernhard said:

But I went ahead and added support for rockmigrations via composer: https://packagist.org/packages/baumrock/rockmigrations

Great, thank you. One more thing you could do is to add git tag for each released version, or at least add the latest one.

[bernhard]

35 minutes ago, Richard Jedlička said:

Great, thank you. One more thing you could do is to add git tag for each released version, or at least add the latest one.

Yeah, maybe I should get used to that. But for me they do not add any benefit... Maybe you could explain why they might be important for others?

[Richard Jedlička]

4 minutes ago, bernhard said:

Yeah, maybe I should get used to that. But for me they do not add any benefit... Maybe you could explain why they might be important for others?

It's because, packagist automaticly creates versions of a package, so you can specify which version you want to install. Without tags you can only install dev-master version (current latest commit) which composer considers as development while tags considers as stable. With tags composer installs the latest tagged version if you don't specify any, but if there is none you have to explicitly specify dev-master if your composer json is not configured to accept it automatically. See my module https://github.com/uiii/ProcessWire-FieldtypePDF as example.

[bernhard]

Thx ? Here you go: https://github.com/BernhardBaumrock/RockMigrations/releases/tag/v0.0.26

I hope I'll get used to tagging and not forget about it in the future ? 

 

[MrSnoozles]

I really love the syntax of the migrations.

I wonder though if RockMigrations could be additionally used like @LostKobrakai described:  to manage the state of templates and fields in a website and create a reproducible deployment.

For that as far as I see two parts would be essential:
  - having a global place to put site-deployments (e.g. site/assets/RockMigrations) where deployments can be put with a timestamp instead of a version
  - a way to run (and track) the run migrations during deployment (maybe as extension of wire shell?)

Having a way to track schema changes between staging and live environments and run them automated in a deployment pipeline would be a huge gain for Processwire.

 

 

[bernhard]

Thx @MrSnoozles 100% agreed. I see two possibilities:

1. Anybody providing PRs
2. Anybody funding/sponsoring further development of RockMigrations by me

I'd love to work more on my modules, but I can't do that all in my spare time ? 

[elabx]

4 hours ago, MrSnoozles said:

really love the syntax of the migrations.

I wonder though if RockMigrations could be additionally used like @LostKobrakai described:  to manage the state of templates and fields in a website and create a reproducible deployment.

For that as far as I see two parts would be essential:
  - having a global place to put site-deployments (e.g. site/assets/RockMigrations) where deployments can be put with a timestamp instead of a version
  - a way to run (and track) the run migrations during deployment (maybe as extension of wire shell?)

Having a way to track schema changes between staging and live environments and run them automated in a deployment pipeline would be a huge gain for Processwire.

 

I think you're talking exactly about the Migrations module from @LostKobrakai and I believe you can use RockMigrations within it without problems. It says it's deprecated but works just fine, I'm using it on latest PW master.

4 hours ago, MrSnoozles said:

Having a way to track schema changes between staging and live environments and run them automated in a deployment pipeline would be a huge gain for Processwire.

I do this exactly with that migrations module having the migrations in version control.

[bernhard]

That's a good idea ? RockMigrations was built so that it can be used easily by other modules. Why not as well by another migration module ? 

[bernhard]

For rapid development it is really great to trigger migrations on Modules::refresh(); As I'm using this technique all the time I've added a new method "fireOnRefresh" to v0.0.27

public function init() {
  $this->rm()->fireOnRefresh($this, "migrate");
}

public function migrate() {
  $this->rm()->createField(...);
}

Another handy update is the trigger() method that I use in combination with custom pageclasses. Custom pageclasses are awesome, but they do not trigger init() and ready() methods by default. That is a pity, because I usually want several hooks to live in the pageclass and not in the module (for example auto-publishing pages or creating random pagenames etc). Now that can simply be done in an autoload module:

// in init()
$rm->trigger("\MyNamespace\MyPageClass", "init");

// in ready()
$rm->trigger("\MyNamespace\MyPageClass", "ready");

I've also updated the example module that uses this technique ? 

[bernhard]

v0.0.30: Working with select options fields got easier today ? 

$rm->migrate([
  'fields' => [
    'yourfield' => [
      'tags' => 'myTag',
      'type' => 'options',
      'inputfieldClass' => 'InputfieldRadios',
      'optionColumns' => 1, // side by side
      'options' => [
        1 => 'Eingangsrechnung',
        2 => 'Ausgangsrechnung',
      ],
      'required' => true,
      'defaultValue' => 1,
    ],
  ],
]);

Setting the "options" as array has not been possible until now.

[MarkE]

Hi @bernhard. Prompted by your comments here , I decided to give this module a go as it looks very useful. One thing I find rather confusing is that you seem to have changed the suggested methodology and it's not entirely clear to me what is now the suggested best practice for using the module.

From what I can see, you are suggesting that users construct their own module (along the lines of your example). Installing this will then run the migration and uninstalling it will do a downgrade. For further migrations, it seems that you are suggesting just adding to this module, not creating a new module. Your previous approach seems to have been more version-based.

I am trying to understand how that works in practice. I currently have 2 migrations that I want to carry out. The dev system reflects the end state of both. I want to do it in 2 stages so that I can throroughly test the first migration before applying the second (although I think there is no dependency, it is better to do them in the same order as on the dev system). 

My plan is therefore to create the first migration module, back up the dev database, load the live database to the dev system, install the migration module to this, then test. If that is OK, then, if I understand correctly, I should amend the migration module to include the second migration and refresh the modules to run it. I would then test again.

Now, what if there is a problem with this second migration and I want to reverse it? Uninstalling the module would presumably undo both migrations (unless I change the uninstall method to be only partial, which seems inconsistent). Sure, in the dev system, I could just revert to a backup, but that would not be a solution if the problems happened in the live environment as (unknown) data will also have been updated - in that case I would need to just be able to reverse the last migration.

So should I put the second migration in a separate module? In which case, this starts to look more like your initial approach (except that the use of the migrate() method allows for a more declarative style). 

Am I clear in explaining my confusion? If so, can you de-confuse me?

By way of background, here is my first migration plan (as a set of human instructions), to give you an idea about what I am trying to do:

1.	Add new fields to templates:
	a.	Property -> operator
	b.	Property -> website2 (new field req’d)
	c.	BookingForm -> propertyPage
	d.	home -> taxYear, dayMonth, month & website2 (with descriptions) 
2.	Amend home template for 
	a.	description of operator (& webmaster?) & website (now allowed)
	b.	re-use of summary field as restricted access field with description/instructions (and resize)
	c.	re-order fields as per dev system
3.	Change title of home to “System”
4.	Add/amend hanna codes for
	a.	ManagementMember
	b.	Operator
5.	Edit cheque payment method to = that on the dev system
6.	Add images and files pages to home
7.	Change title of home
8.	Re-purpose comparison field as “mailHeaders” and add to StandardMails template
9.	Remove bodyTop field from StandardMails template – change body to 10 rows, summary to 3, footer to 2, update descriptions & notes in the template and update instructions in the page

I think I can see that there are methods for most of that in your module, but I'm not sure about the hanna codes. The php in all my hanna codes is identical (as it just passes data to a separate script), but obviously attributes etc. are different. I don't think RockMigrations handles these, but they have convenient export/import features so that's not too much of an issue.

The second migration is 

1.	Use findPages to detect all pages with BookingStatus in a body field (i.e bodyTop etc. as well as body – any field that can take a hanna code) – approx. 15 pages. Make appropriate changes.
2.	Change the refNumber for cancelled statuses (2 pages).

I can probably turn these human instructions into php (although it may be quicker just to do it in the admin, but then I definitely do not have a quick 'downgrade option!).

[MarkE]

I'm assuming from the 'like' that I am thinking on the right lines. I've started building the migration module and it's coming along nicely - a dream to test it as you go along.

Now for a bit of heresy ?. I wanted to migrate my Property template. I added the field website2, then I wanted to define the new state of the template with the new field, the amended widths etc. I couldn't quite remember what I changed (although not too hard to work out in this instance) but why should I? - surely I could just declare the whole thing as set out in Tracy's fieldgroupContexts for the template. But now followed a bit of awkward copying and pasting from Tracy's Request Info panel. Why not have a php script to interrogate the current (dev) state and create the required array...?

Oh dear - I started by building a simple migration and now am in danger of disappearing down a rabbit hole... I suspect this could become a bigger issue than just this module. I see that @Kiwi Chris has had similar thoughts over on the thread that sparked my interest in this.

[bernhard]

9 hours ago, MarkE said:

I'm assuming from the 'like' that I am thinking on the right lines.

I have not had the time to read your post carefully and give an educated answer ? 

9 hours ago, MarkE said:

Why not have a php script to interrogate the current (dev) state and create the required array...?

Why not define those changes directly in your code from the very first moment? I know that feels complicated and cumbersome, but you'll need the same things over and over again (mainly being "type", "columnWidth", "label", "tags") and you'll get used to that and in the end be at least as quick as doing it by clicking around the admin. In more complex scenarios you'll likely be even quicker (imagine changing the "columnWidth" of multiple fields to rearrange your setup... that means a lot of clicks... on the dev-setup and then a lot of clicks and brain work to replicate that on production; using code you can simply change all widths, save, refresh).

Back to your first post...

22 hours ago, MarkE said:

One thing I find rather confusing is that you seem to have changed the suggested methodology and it's not entirely clear to me what is now the suggested best practice for using the module.

Yes, that's a good observation. The first approach was inspired by Lostkobrakai's module that has a concept of "one single truth/place for all migrations". I just wanted to make the syntax easier to grasp and use. I see the benefits of that approach. Even more because I have improved a lot as developer and I'm packing almost anything into modules or reusable components. On the other hand, that approach has a huge drawback: You end up with lots of migrations and you can't easily grasp what's going on in the big picture.

My new approach is a lot more practical (though might be less technically "correct"): You put your setup into a dedicated migrate() method of your module that represents the state/setup of this module or pageClass. Then you see at one glance all fields, templates and pages that belong to that logical piece of your project and with a single modules refresh you get that setup on your system - and with a git push and refresh you also get that setup on your live system (or any other).

There are rare cases where the execution order of migrations is important. But it turned out that this is really rare on the one hand and almost impossible across different logical parts (eg two different modules). And even if that happend... You have the same problem with regular PW modules or any other kind of software dependency issues...

In 95% of the cases migrations are all about adding fields, creating pages that hold data, changing parameters (like labels). All that is really as easy as changing properties of the migrate([...]) options array. Sometimes you might realize that you've created a field that you don't need any more. Then I simply use this technique:

// in one of your modules or page classes

public function cleanup() {
  $this->rm()->deleteField('myoldfield');
}

public function migrate() {
  $this->cleanup();
  $this->rm()->migrate([
    'fields' => [
      'mynewfield1' => [...],
      'mynewfield2' => [...],
    ],
    'templates' => [...],
  ]);
}

public function rm() {
  return $this->wire->modules->get('RockMigrations');
}

I even use class constands now instead of string values - that has a huge benefit that I get better readability in my code (syntax colors), it's impossible to make typo's and I get code completion wherever I am in my project:

You see in this example: The properties that are needed are really easy:

• label --> just type a string
• icon --> just choose one of the fontawesome icon-names (I tend to look them up here https://fontawesome.com/v4.7.0/icons/ but I guess ther should also be a plugin for autocomplete?)
• collapsed state --> autocomplete again ? 
  

If I come up with a property that I don't know, I do that manually on that field, look up the property in tracy, apply the code to my module/pageClass, save it, refresh and check if everything works, commit and continue with my work ? 

PS: In the screenshots above you see a migration that was applied when the fields already existed in the system (I built that before I had the joy of RockMigrations). That's why some other properties are not defined (like type, tags,...). But it shows also how flexible and easy to apply RM can be. One might call it "progressive" ?

Does that bring some light into the dark?

[MarkE]

Thanks @bernhard, that's really helpful. It is prompting me to re-think my whole approach to PW development. I had already moved to wholesale use of template page classes; If I understand you correctly, you are now suggesting that I use the Page Class to hold the complete template definition via migrate(). And that I use that rather than the admin tools to maintain the template. Your first comment 

40 minutes ago, bernhard said:

Why not define those changes directly in your code from the very first moment?

makes even more sense in that context. You will appreciate that, of course, the original app was not developed in that way - hence my need to copy and paste out of Tracy (a quick way of doing that would make moving onto your way of doing things much easier). I will read your post again and try a few things to make sure I have an approach that works for me.

Meanwhile a couple of things occur to me:

1. The way RockMigrations has eveolved (and maybe it's still evolving...) has meant there there is not a consistent documentation of the whole approach and the related API. This is not a criticism - I appreciate that you are just sharing something and are not obliged to document it completely and also that, while it is evolving, documentation is a painful overhead. However, being forgetful, I will need to document something for myself, so happy to share if it is useful.
2. Exploring this module and reading your comments has raised a whole bunch of questions in my mind about the direction of PW. There has been much discussion lately about front-end enhancements, but I think attention should also be paid to the back-end and in particular to the issue of maintainability which your module addresses. I think there is a place for both UI-based development via the admin and code-based development like you suggest (more like a trad PHP framework). In particular the UI-based tools are an easy entry point for new users and for prototyping, so a route from that to the code-based approach would be helpful (hence my comments re Tracy copy & paste). I'm still quite a PW newbie and a fairly inexperienced PHP user, so I'm a bit nervous about putting this into the PW roadmap discussion without having a clear idea of the implications, but it seems like that is where it belongs. If you or other key players such as @ryan or @adrian want to pick up on this then that would be great.

[bernhard]

1 hour ago, MarkE said:

If I understand you correctly, you are now suggesting that I use the Page Class to hold the complete template definition via migrate()

That's exactly how I'm developing PW now. Everything related to a pageClass is in the pageClass - even hooks. That's by default not possible because we have no init() or ready() in pageClasses. That's why there is the trigger method: https://github.com/BernhardBaumrock/RockMigrations/blob/88faf3f13844de3e4a8a4ae2968dc45535778659/RockMigrations.module.php#L346-L362 There's also loadClasses here https://github.com/BernhardBaumrock/RockMigrations/blob/88faf3f13844de3e4a8a4ae2968dc45535778659/RockMigrations.module.php#L1851-L1863

Unfortunately that's a little mess. That methods have been quick additions while finding the right way to do it. I didn't know how to use PW's classloader back then and I even didn't know about  spl_autoload_register... But it does not matter too much to me right now as long as the pageClass's init() gets called on init() and ready() - if needed - on ready(). Think of it as an autoload pageClass. Not sure about the performance impact of that, but I guess it's minimal.

The benefit is that I have everything in one file that belongs together. All hooks, all helper methods for markup output, whatever. No more hook chaos in ready.php, no more problems moving updates from dev to live and of course having all changes documented in GIT ? 

1 hour ago, MarkE said:

Thanks @bernhard, that's really helpful. It is prompting me to re-think my whole approach to PW development.

I guess your approach is a lot more common than mine... that's why migrations get a lot less attraction than they'd deserve in my opinion...