RockMigrations 🚀🔥 The Ultimate Automation and Deployment-Tool for ProcessWire

[Ivan Gretsky]

3 hours ago, teppo said:

I'm considering RockMigrations for a new project

Hey @teppo! I am running circles around this module for a long time. And last week did install it on a dev env for the 1st time. I am not quite sure how to integrate it in my workflow. And not sure my workflow fits it (not the other way around))) Please write back about your experience with RockMigrations if you progress with using it. Your opinion and feedback are a really valuable!

[dotnetic]

What do you struggle with @Ivan Gretsky? I use RockMigrations for every project right now. You got version control for your fields and templates, and even more.

So if you are using git and develop new features, it is of great use because it adds the needed fields/templates when switching branches. Also in conjunction with RockFrontends livereload feature, this is a lot of fun. You add a field in the migrate.php (or somewhere else where you need it, like a custom page class) and the page in the admin autmatically shows the newly added field.

This made my workflow so much quicker.

I don't use the deploy feature (github actions) because I got my own github actions, so I can not say anything about that.

[dotnetic]

Another thing is, you can re-use custom page classes in new projects and have their fields/templates created automatically. Develop them only once, copy them over to a fresh PW install, and there you go.

[Ivan Gretsky]

I cannot say that I am stuggling... But those automatic things make me a bit nervous. I would much prefer to run those migrations manually.

Another thing is I need to work in the environment where some changes to fields and templates are still made in the admin. And  that red notification there is no way to deal with that,

 

[bernhard]

On 10/28/2022 at 1:52 PM, Ivan Gretsky said:

But those automatic things make me a bit nervous. I would much prefer to run those migrations manually.

You can do that. RockMigrations will only do what you tell it to do.

On 10/28/2022 at 1:52 PM, Ivan Gretsky said:

Another thing is I need to work in the environment where some changes to fields and templates are still made in the admin.

You can also do that and sometimes I do it myself.

On 10/28/2022 at 1:52 PM, Ivan Gretsky said:

And  that red notification there is no way to deal with that

It's just a reminder that if you apply changes that are somewhere in code of a migration your changes will get overwritten. If you apply changes via GUI that are not set in any migration than you'll be fine and your manual changes will be kept.

That indicator could be improved I guess. For example it could only appear on fields or templates that received changes from a migration. And changes done via RockMigrations could be listed instead of the generic warning. If that is an important feature for you (or anybody else) I'd be happy to merge any PR in that direction and provide all the help needed.

[bernhard]

The latest version of RockMigrations comes with a console on the module's settings page. What does that mean?

It is now easier than ever to create custom "site-profile" like installation scripts that you can just copy and paste into that console. Add or remove the lines that you need or don't need, check the checkbox to run code on save and hit the save button!! ?

[Kiwi Chris]

On 10/29/2022 at 1:17 AM, bernhard said:

That indicator could be improved I guess. For example it could only appear on fields or templates that received changes from a migration. And changes done via RockMigrations could be listed instead of the generic warning. If that is an important feature for you (or anybody else) I'd be happy to merge any PR in that direction and provide all the help needed.

I think this could work both ways. One of the things I really like is that it's possible to use the admin UI to design your fields and pages, but RockMigrations then gives you the code generated to copy and paste into your migrations file.

If you start prototyping using the UI, and intended to add it to a migration but forgot to, having an indication that a field or template isn't included in a migration is just as useful as indicating that it is. This would be particularly useful when adding RockMigrations to an existing site, as it makes it easier to see quickly what has been see up to be managed by migrations and what hasn't been.

[bernhard]

Yeah I think that would be a good addition. I'm happy to help whoever wants to tackle that ? 

[Spinbox]

Asked this question privately to Bernhard, his reaction included

I have been digging into RockMigrations and I'm trying to learn the workflow.

I have the following MagicPage.

<?php namespace ProcessWire;

use RockMigrations\MagicPage;

class ProjectPage extends Page {
	
	use MagicPage;

	public function migrate(): void {

		$rm = modules('RockMigrations');
		
		$rm->migrate([
			'fields' => [
				'project_short_description' => [
					'type' => 'FieldtypeTextarea',
					'label' => 'Beschrijving',
					'notes' => 'Korte beschrijving zichtbaar op andere pagina\'s',
					'rows' => 5,
					'columnWidth' => 50,
					'icon' => 'align-left',
					'description' => '',
					'inputfieldClass' => '',
					'contentType' => 1,
				], 
				'project_description' => [
					'type' => 'FieldtypeTextarea',
					'label' => 'Inleidende beschrijving',
					'rows' => 5,
					'columnWidth' => 50,
					'icon' => 'align-left',
					'description' => '',
					'inputfieldClass' => '',
					'contentType' => 1,
				],
				'project_client' => [
					'name' => 'project_client',
					'label' => 'Opdrachtgever',
					'type' => 'FieldtypeText',
				],
				'project_subtitle' => [
					'label' => 'Subtitel',
					'type' => 'FieldtypeTextarea',
					'rows' => 2,
				], 
				'project_state' => [
					'label' => 'Status',
					'type' => 'FieldtypeSelect',
					'select_options' => '1:=In voorbereiding
					  2:=In uitvoering
					  3:=Gerealiseerd',
					'columnWidth' => 33.33334,
				],
				'project_number' => [
					'label' => 'Aantallen',
					'type' => 'FieldtypeText',
					'columnWidth' => 33.33334,
				],
				'project_partners' => [
					'label' => 'Partners',
					'type' => 'FieldtypeText',
					'columnWidth' => 33.33334,
				],
			],
			'templates' => [
				'project' => [
					'fields' => [ 
						'title',
						'project_short_description',
						'project_description',
						'project_client',
						'project_subtitle',
						'project_state',
						'project_number',
						'project_partners',
					]
				],
			]
		]);

	}

}

I have removed all the fields and templates that are in this migration. I was expecting the migration to execute when updating the migration inside the MagicPage and create all the fields and the template. This however works only if I add to migrate.php:

$rm->migrate(['templates' => ['project']]);

Is it possible to trigger the RM via MagicPage only? Or is this by design (Now that I think about it I can imagine you may want to have the MagicPage file in your project but not let it trigger)?

------

@bernhard's reply:

Quote

 

If it is a magicpage it should trigger the migrate() automatically, but only if the file is changed.

You can easily debug/visualise that by adding a tracy bd('test1'); in the migrate() method and once you update that to 'test2' you should see the bar dump in the debug bar.

If you don't change anything the migrate() will not trigger.

 

-------

I have tested this, but in the case I have not created a 'project'-template yet, this will not trigger. I believe the Custom Page Class will only be triggered if you actually have a template with that name. As soon as I create a project template the migrate will trigger (which is fine for me). 

[bernhard]

2 hours ago, Spinbox said:

I have tested this, but in the case I have not created a 'project'-template yet, this will not trigger. I believe the Custom Page Class will only be triggered if you actually have a template with that name. As soon as I create a project template the migrate will trigger (which is fine for me). 

Yes, that is true! Sorry I didn't understand that the template of your pageclass does not exist yet! MagicPages can only be loaded if the template exists, therefore you need to make sure that the template is being created before you can actually use the migrate() method.

RockMigrations has the "migratePageClasses()" method for that: https://github.com/baumrock/RockMigrations/blob/d429872cbcb6d658352844fb612ee4a10baa57a2/RockMigrations.module.php#L2311

And it's documented in the WIKI how to do it (also setting parent-child relationships which is a common need): https://github.com/baumrock/RockMigrations/wiki/Ship-your-Module-with-Custom-Page-Classes

Thx for the question, I've made the docs a little clearer about that ? 

[snck]

@bernhard I was a little confused whether RepeaterMatrix fields are (still) supported or not. I found this discussion concerning RockMigrations1 and was somehow expecting that the support for RepeaterMatrix fields was also ported to RockMigrations(2). Are RepeaterMatrix fields still supported and if not, do you have any plans to change that?

Cheers,
Flo

[gebeer]

@snck I am currently working on implementing Repeater Matrix support. It is basically working but needs some more thorough testing. ATM the setMatrixItems() method seems to be working well. If you like, you can grab a copy from https://github.com/gebeer/RockMigrations/tree/repeatermatrix and test it out.

This is how you create a Repeater Matrix field:

// first create the blank field
$rm->createField('repeater_matrix_test', 'FieldtypeRepeaterMatrix', ['label' => 'Product Content Blocks', 'tags' => 'products']);
/**
 * Set items (matrixtypes) of a RepeaterMatrix field
 *
 * If wipe is set to TRUE it will wipe all existing matrix types before
 * setting the new ones. Otherwise it will override settings of old types
 * and add the type to the end of the matrix if it does not exist yet.
 *
 * CAUTION: wipe = true will also delete all field data stored in the
 * repeater matrix fields!!
 */
$rm->setMatrixItems('your_matrix_field', [
    'foo' => [ // matrixtype name
        'label' => 'foo label', // matrixtype label
        'fields' => [ // matrixtype fields
            'field1' => [
                'label' => 'foolabel', // matrixtype field options
                'columnWidth' => 50, // matrixtype field options
            ],
            'field2' => [
                'label' => 'foolabel', // matrixtype field options
                'columnWidth' => 50, // matrixtype field options
            ],
        ],
    ],
    'bar' => [ // matrixtype name
        'label' => 'bar label', // matrixtype label
        'fields' => [ // matrixtype fields
            'field1' => [
                'label' => 'foolabel', // matrixtype field options
                'columnWidth' => 50, // matrixtype field options
            ],
            'field2' => [
                'label' => 'foolabel', // matrixtype field options
                'columnWidth' => 50, // matrixtype field options
            ],
        ],
    ],
], true);

Still need to test the other methods for removing/adding items etc. But this one should work. At least it was working in my tests.

When testing is completed I will make a PR so @bernhard can implement.

[Kiwi Chris]

A quick question before I go ahead and set up migrations files:

I like using the ProcessWire UI to edit my field and template properties, and RockMigrations gives me all the code to copy and paste into a migrations file, so this is nice and easy, but I want to make sure on my development instance that I don't accidentally overwrite field or template definitions by triggering a migration when I've yet to copy and paste updates from those fields into the migrations file.

I do want the migrations to run automatically on the deployed site I think though, as the migrations file(s) will only get updated on the deployed site when they're ready to run.

I think I'd prefer to include the migrations in page classes, as that keeps all the definitions of an object in one place and nice and modular, just I don't want to accidentally trigger these on my local development environment before I have all the fields and templates modified with any changes I've done via the UI.

What's the best way to handle this scenario?

[bernhard]

Hi @Kiwi Chris I've added a config setting to prevent migrations if you don't want them to be triggered automatically:

$config->noMigrate = true;

https://github.com/baumrock/RockMigrations/#running-migrations

For adding this setting only to your local dev setup see https://processwire.com/talk/topic/18719-maintain-separate-configs-for-livedev-like-a-boss/

[Kiwi Chris]

13 hours ago, bernhard said:

$config->noMigrate = true;

This is perfect.

I've thought of an enhancement to RockMigrations that would be useful because I'm lazy. 

In the showCopyCode method, instead of copying and pasting, an additional option would be an 'export' button, and a path field.

I'd still need to write a migration file, as the order of field and template migrations is important, but if I simply list the fields and templates as includes in order, then it would make the migration file more concise, and I could still trigger a migration by updating version details of file, but without having to keep copying and pasting into a file.

eg, if I want the definition for myTextField to be associated with ProcessHelloWorldModule, then I could set a path site/modules/ProcessHelloWorldModule/fields/ and dump the definition there by button click.

I've been having a play and here's a mockup although it doesn't do anything yet.

I'm not sure whether it actually needs its own save button, or whether to hook into saving the field and save the RockMigrations code to the path specified.

I've never worked with modifying field/template editing before, so I'm not sure how to persist the save path.

Of course I can do this manually, by copying and pasting each field and template definition into its own file but I think this might be a time saver for people who like to use the UI, but also want to use migrations.

[bernhard]

Hey @Kiwi Chris thx for your input, that's an interesting idea!

I've often thought about how we could do something similar. The problem is to define what should happen automatically and what manually. The distinction is the key. Sometimes it might be good to have things happen automatically, sometimes the opposite would be preferable. That's why it's 100% manual at the moment.

Defining a path to save the code could be an option I guess. The storage/save portion of the problem is trivial. Instead of showing the code in the backend we could easily store it somewhere. But how would you suggest using that file then?

Copy&pasting code is easy:

$rm->createField('foo', 'text', [
  'label' => 'my foo field',
]);

How would that look like when having a code file?

$rm->createField('foo', '/site/modules/MyModule/RockMigrations/fields/foo.json');

While that looks good on first sight I'm not sure it will really work. Maybe it could work for fields, but for templates it might not:

$rm->createTemplate('foo', '/site/modules/MyModule/RockMigrations/templates/foo.json');

What if that file had a reference to a field that does not exist? One would have to make sure that this field exists before the createTemplate is called. Clear what I mean?

When writing migrations manually these problems are likely less of an issue (I guess), because while writing that migrations code you should already realise if you are referring to something that is not yet created.

I've also thought about a solution that copies all field and template setups to a dedicated place (like /site/assets/RockMigrations/fields/... or /templates/... )  and on migrate() it first creates all those fields and templates and then does another run that actually applies all settings. That's what the migrate() method does behind the scenes.

Maybe it helps if you provide a very detailed step by step use case / workflow how you think that concept could work and how you think we could solve the problem of non-existing references.

[Kiwi Chris]

5 hours ago, bernhard said:

The problem is to define what should happen automatically and what manually.

That's a significant issue, and I tend to think it's better to have migrations generated manually rather than fully automated, but I think possibly more automation is possible.

Here's my first ever migrations file, that works nicely for me.

//Modules
$rm->installModule(
				"FieldtypeLeafletMapMarker",
				"https://github.com/madebymats/FieldtypeLeafletMapMarker/archive/PW3.zip"
);

//Fields
include __DIR__ . '/fields/fields.php';

//Templates
include __DIR__ . '/templates/Discipline.php';
include __DIR__ . '/templates/Member.php';
include __DIR__ . '/templates/Show.php';
include __DIR__ . '/templates/ProductionRole.php';
//Web Templates
//include __DIR__ . '/templates/contact.php';
include __DIR__ . '/templates/ProductionRole.php';
include __DIR__ . '/templates/location.php';
include __DIR__ . '/templates/pastShows.php';

$rm->createPage('Website', 'website', 'basic-page', 'home');
$website = $this->wire->pages->get('name=website');
$rm->createPage('Past Shows', 'past-shows', 'pastShows', $website);
$rm->createPage('About Us', 'about-us', 'basic-page', $website);
$rm->createPage('Contact', 'contact', 'location', $website);

In each of those template definitions, I have:

$rm->createTemplate('...');
$rm->setTemplateData(
...
);

I did combine the field definitions into one file, but I could have done them individually.

I still have full manual control over the order of migrations, but my migrations file itself is pretty small, and each template definition file is self-contained, and I think it makes the code cleaner.

It's not as efficient maybe as running $rm->migrate with all the templates in one big array but it works, and I can control the order.

It made me think, if I'm using this sort of structure, then the getCode method of RockMigrations, instead of being used just to display the code on screen, could build the definition files for me.

A way to have some manual control if definitions are combined in a file would be to have some sort of Migrations build file for a module, which specifies the fields and templates in the order that they need to be processed. In my module migrations file, I've specified the order manually, but I've kept the definitions separate.

The way I'm using this is in association with modules, so I wonder whether the way to do it is to hook into the module UI, check whether the module has a migrations build file, and if it does, offer a button to generate the module migration based on the specification in the build file?

[bernhard]

3 minutes ago, Kiwi Chris said:

Here's my first ever migrations file, that works nicely for me.

Congrats ?

3 minutes ago, Kiwi Chris said:

$rm->createPage('Website', 'website', 'basic-page', 'home');
$website = $this->wire->pages->get('name=website');

This should be possible like this:

$website = $rm->createPage('Website', 'website', 'basic-page', 'home');

 

6 minutes ago, Kiwi Chris said:

I did combine the field definitions into one file, but I could have done them individually.

I still have full manual control over the order of migrations, but my migrations file itself is pretty small, and each template definition file is self-contained, and I think it makes the code cleaner.

It's not as efficient maybe as running $rm->migrate with all the templates in one big array but it works, and I can control the order.

It made me think, if I'm using this sort of structure, then the getCode method of RockMigrations, instead of being used just to display the code on screen, could build the definition files for me.

A way to have some manual control if definitions are combined in a file would be to have some sort of Migrations build file for a module, which specifies the fields and templates in the order that they need to be processed. In my module migrations file, I've specified the order manually, but I've kept the definitions separate.

I wonder if it could make sense to have all these migration files in one single folder, eg /site/assets/RockMigrationsConfig/

This could then be called with a single $rm->migrateConfig("/site/assets/RockMigrationsConfig") call. It could take care of creating all fields and template first and then run the migrations that set all settings.

Your code could then become something like this:

//Modules
$rm->installModule(
  "FieldtypeLeafletMapMarker",
  "https://github.com/madebymats/FieldtypeLeafletMapMarker/archive/PW3.zip"
);

$rm->migrateConfig("/site/assets/RockMigrationsConfig");

$website = $rm->createPage('Website', 'website', 'basic-page', 'home');
$rm->createPage('Past Shows', 'past-shows', 'pastShows', $website);
$rm->createPage('About Us', 'about-us', 'basic-page', $website);
$rm->createPage('Contact', 'contact', 'location', $website);

What do you think?

[Kiwi Chris]

1 hour ago, bernhard said:

I wonder if it could make sense to have all these migration files in one single folder, eg /site/assets/RockMigrationsConfig/

This could then be called with a single $rm->migrateConfig("/site/assets/RockMigrationsConfig") call. It could take care of creating all fields and template first and then run the migrations that set all settings.

Something like this could work.

There are two possible scenarios I can think of.

A site profile migration or a module migration.

In either case, it's still necessary to be able to manually specify the order of the fields and templates because of possible dependency issues, but it should be possible to generate the field and template definitions (for those who want to, rather than coding them). 

 

[Alpine418]

Hi.

I have tried out RockMigration and have a few questions (and still dont get it, I how it should be implemented in a running website on dev and prod stages).

I have a working website. Can I generate a fully migrate.php file or do I need to create it manually with the code hints of each field/template? Last one would be a real pain if there are many fields.

If I have a field added and want to remove it later on in development process. Then how can I different the state of the field in the migrate.php? Top to down, first added and then removed in one run? But how would the migrate.php looks then?

Are there any advanced examples of migrate.php for learning the basics and inspiration? Or advanced docs?

Thanks.

[bernhard]

Upate: I've created a video for you:

2 hours ago, rjgamer said:

Can I generate a fully migrate.php file or do I need to create it manually with the code hints of each field/template?

Why do you want to do that? You don't need to ? 

2 hours ago, rjgamer said:

Last one would be a real pain if there are many fields.

Correct. That's why I didn't build it that way.

2 hours ago, rjgamer said:

If I have a field added and want to remove it later on in development process. Then how can I different the state of the field in the migrate.php? Top to down, first added and then removed in one run? But how would the migrate.php looks then?

// content of migrate.php @ 2023-01-01
// create the foo field and add it to the foo template
$rm->createField('foo', [...]);
$rm->addFieldToTemplate('foo', 'footpl');

// content of migrate.php @ 2023-01-10
// remove foo field and create bar field instead
// "true" makes it "quiet" - if not set it will write a log on every
// following migration that says "Field foo not found"
$rm->deleteField('foo', true);
$rm->createField('bar');
$rm->addFieldToTemplate('bar', 'footpl');

// content of migrate.php @ 2023-01-16
// refactor to array syntax
// that's nicer because you can instantly sort fields on the template
// in the example we add 'bar' on top of 'title'
$rm->deleteField('foo', true);
$rm->migrate([
  'fields' => [
    'bar',
  ],
  'templates' => [
    'footpl' => [
      'bar',
      'title',
    ],
  ],
]);

 

2 hours ago, rjgamer said:

Are there any advanced examples of migrate.php for learning the basics and inspiration? Or advanced docs?

https://github.com/baumrock/RockMigrations/wiki/Ship-your-Module-with-Custom-Page-Classes This is an advanced example.

[Alpine418]

Thank you! Will give it a try again.

Update: Sorry, I was in hurry. I just saw you made a video for me. This is like VIP support ?

[Alpine418]

Hi. Again me with a question.

How does the migration of modules work with RockMigration? There is a watcher, but it only watches files and no database changes.

Example: I've installed a new module installed with a few custom configurations on my dev environment (e.g. the Duplicator module). Now I want to have the module installed on prod environment too. Does RockMigrations with watcher really catch the module installation and configuration or what is the best way to do handle a new module installation and configuration?

Thanks for your support, and no... I don't really need a first class VIP support video from @bernhard again. But you are free born of course?

[bernhard]

// site/migrate.php
$rm->installModule('YourModule', [
  'fooSetting' => 'fooValue',
  'barSetting' => 'barValue',
]);

See https://github.com/baumrock/RockMigrations/blob/main/profiles/default.php for inspiration. Please let me know if anything is unclear and can be improved.

[Kiwi Chris]

I'm not sure whether it's intentional or not, but I started experimenting with deploying a site setup to a clean, blank copy of ProcessWire with RockMigrations

I dumped a copy of RockMigrations and a custom module with a migration file in the format MyModule/MyModule.migrate.php

I installed RockMigrations via the admin UI, and was wondering why it was taking so long. Afterwards I checked, and found it had gone ahead and installed all the modules, fields, files, and created pages in the custom module's migrate file, even though the module itself was not yet installed.

I'm not sure whether that's intended behaviour? If it is, I need to make sure I don't upload a module with a migrate file unless I want the migration to run as soon as I upload it.

My assumption was that RockMigrations would start watching and running migrations in a module's directory once the module is installed, but possibly I assumed wrong.

Apart from being surprised that the module's migration file ran before the module was installed, it was very nice to see a whole app with multiple fields, templates, modules, and pages constructed before my eyes, and it's way more modular than exporting site profiles, since I can build a base functionality in one migration file and then add components in others.