RockMigrations 🚀🔥 The Ultimate Automation and Deployment-Tool for ProcessWire

[bernhard]

I have just released version 2 of RockMigrations:

GitHub: https://github.com/baumrock/RockMigrations
Modules Directory: https://processwire.com/modules/rock-migrations/

Please star the module on GitHub if you like it ? 

Are you unsure if RockMigrations is the right tool for you?

 

On 3/25/2023 at 2:44 PM, Jonathan Lahijani said:

RockMigrations is a must-have for any ProcessWire developer who wants to define fields and templates with code. This functionality elevates ProcessWire to the level of a true web application framework and incorporating RockMigrations into my workflow has become essential.

@Jonathan Lahijani in a very nice PM ? 

On 9/11/2022 at 9:08 AM, gebeer said:

In the past I had used the old version of RockMigrations for some simple tasks like adding fields and templates which turned out to be huge time saver. But only just recently I started to use the new version and to discover the possibilities of this tool in more depths. And what should I say, I am really amazed. Once I had grasped the core concepts and, with @bernhard's help, had learned how to use it in a more structured way and in different contexts, it turned out to to be an even bigger time saver.

Read the full post here

On 3/20/2023 at 11:10 PM, gornycreative said:

My main comment, though, is that the marriage of page classes and migrations is truly groundbreaking in terms of modular design and in particular the ability to bring over single files that can instantly put in (or remove) scaffolding for demos. This has been AWESOME for live brainstorming and troubleshooting or providing clients with examples of how extensible the system is and how quickly modifications and improvements to functionality can be built out.

This has made me more excited about using RockMigrations - I was a bit on the fence when the automation was centralized in migrate.php because of how unwieldly automation scripts can get in other applications I use - but bringing things that involved scaffolding into their respective page classes allow me to really just use migrate.php for mod installations, configuration, general hooks and other core modification processes.

This is way easier to navigate, share and especially train on. This method is very easy to coach new devs on also.

Read the full post here

On 3/10/2023 at 10:52 AM, dotnetic said:

I use RockMigrations every day and love it, as I stated often before. It isn't as hard as it looks at first. Just try it out and if you should stumble over something, then just ask for support here in the forum. RockMigrations saves much time and makes it easy to develop features in your dev environment and then when the feature is finished push the changes to the live server with migrations being executed automatically, which creates all fields, templates and even pages and contents. Could live without it, but that would be a sad life.

So.... start using it now!

Read the full post here

QuickStart

The example code uses bd() calls for dumping data. You need TracyDebugger installed!

Put this in your site/migrate.php

/** @var RockMigrations $rm */
$rm = $modules->get("RockMigrations");
bd('Create field + template via RM');
$rm->createField('demo', 'text', [
  'label' => 'My demo field',
  'tags' => 'RMDemo',
]);
$rm->createTemplate('demo', [
  'fields' => [
    'title',
    'demo',
  ],
  'tags' => 'RMDemo',
]);

Reload your site and you will see the new field and template in the backend and you'll see the message in the tracy debug bar.

[bernhard]

I've just implemented a new way of how watching files work and bumped the version to 1.0.0

As from version 1.0.0 the module will not run ALL migrations any more but only the ones that either have the "force" property set or that have changed since the last migrations run.

I'm working on a site with lots of RockMatrix blocks and all blocks have their own migrate() method. Using RockFrontend I get livereload as soon as I change a watched file, but then the reload took over 9 seconds because I had so many migrations going on. Now the new version does only migrate changed files and forced files, which brings down the duration for the reload to around 1 second ?

When using migrate() from the CLI it will still trigger all migrations, even for unchanged files.

That update will especially be interesting for you @dotnetic I guess? ? 

This is how the log looks like (HomePage was changed and RockMatrix is forced, so these two files are the only ones being triggered):

[pwired]

Modules Directory link does not work (22:10 gmt)

[bernhard]

8 hours ago, pwired said:

Modules Directory link does not work

I guess it just needs to get published by ryan

[MarkE]

The new version looks good @bernhard and I am very tempted to give it a proper try. However, I have been using my own ProcessDbMigrate module successfully on a number of projects and it is serving me pretty well (and the change in approach would be quite radical). Although the two are quite different in methodology, there are some similarities in that my json files have similarities to your yaml files (btw, not quite sure what advantages yaml has over json in this context - json has coped so far).

My module is still very much alpha as something this complex does need extensive testing and bugs do crop up (as well as the need to extend to handle additional field types etc. - the recent version 0.1.5 has been updated to include FieldtypeRepeaterMatrix). Sometimes I think about re-engineering it to use RockMigrations methods rather than the current native ones, but that does lead to some head-scratching.

For example, my module will handle uninstallation of migrations (provided they have not been conflicted by subsequent changes) - I don't see how RockMigrations does this (reverting the code would leave the database unchanged?).

Also I can attach hooks to run on installation (and uninstallation) to handle any data changes related to the migrations (e.g. say field 'name' is split into 'first_name' and 'last_name', the code can populate the new fields when the installation is run). I assume that RockMigrations can handle that too (at least for the forward install), but not quite sure how.

I know you have commented in the past that you don't like my approach (although I'm not sure of the precise reasons**) but I think it is a valid alternative and I would like to somehow get the best of both worlds.

** I can see that using Git to handle conflicts has some advantages. My module has to include its own conflict management which works but has not been tested in a multi-developer environment. Mind you, I can envisage semantic conflicts that might arise which Git would not spot - separate modules changing the same field for example.

[bernhard]

14 minutes ago, MarkE said:

Also I can attach hooks to run on installation (and uninstallation) to handle any data changes related to the migrations (e.g. say field 'name' is split into 'first_name' and 'last_name', the code can populate the new fields when the installation is run). I assume that RockMigrations can handle that too (at least for the forward install), but not quite sure how.

Hey @MarkE thx for your message! I'll answer in detail soon. Could you please describe exactly what you mean here? Best with a specific example from the very beginning until the very end. For someone that is not really familiar with your module please ? What do you do in code, what do you do in the GUI? Where? When? Why?... Thx

[MarkE]

8 hours ago, bernhard said:

Could you please describe exactly what you mean here?

In case you hadn't realised, there is extensive documentation of the module at https://metatunes.github.io/DbMigrate/help.html. So you don't need to install and use the module to get an understanding of what it does. Specifically as regards running code on migrations, see the following sections: https://metatunes.github.io/DbMigrate/help.html#snippets and  https://metatunes.github.io/DbMigrate/help.html#hooks.

I'm happy to supply a full soup-to-nuts example if it helps, but that will take a little time. Perhaps reading the help file and reviewing this (completed) example might be explanantion enough. (Note that this example does not include ready.php code for uninstalling as I had fully tested everything and was confident I didn't need to write it. If for some reason it was needed, I could have modified ready.php to hook before and after uninstallMigration() in a similar manner).

Let me know if you want a fuller example; also if anything in the help file is not clear as I'll gladly improve it.

[MarkE]

 Confused @bernhard? I’m not sure what is confusing. Did you read the manual? I’m happy to try and clarify if I know where the problem is. 

[bernhard]

On 9/2/2022 at 1:36 AM, MarkE said:

Although the two are quite different in methodology, there are some similarities in that my json files have similarities to your yaml files (btw, not quite sure what advantages yaml has over json in this context - json has coped so far).

I'm not using yaml at all. That was just a quick test because many were asking for it. I'm still not convinced. For the same reasons as why I'm not convinced with your approach (at least how I'm understanding it). I guess the reason why our files are similar is no surprise, because our modules describe what PW has to do and of course the data structure of PW is the same for your module and mine.

On 9/2/2022 at 1:36 AM, MarkE said:

For example, my module will handle uninstallation of migrations (provided they have not been conflicted by subsequent changes) - I don't see how RockMigrations does this (reverting the code would leave the database unchanged?).

https://processwire.com/talk/topic/26635-add-better-configuration-for-fields-and-templates-and-make-them-version-controllable/?do=findComment&comment=220791

The point here is: I know that when it comes to migrations the "standard" way of doing it is to provide an upgrade() and a downgrade() code. I've started that way too (back in Jan/2019 https://github.com/BernhardBaumrock/RockMigrations/blob/a2b90ba4fc1aa2b4a6c4e1dc976ca644ae492888/.RockMigrationsDemoModule/RockMigrations/0.0.1.php )

Since then it turned out that writing downgrade methods is just a waste of time. I've never - really: never - had the need for a downgrade since then. Usually when developing and when reverting thins a simple db-pull from the staging system is enough and all the changes are gone. No need for a downgrade. If changes have already been pushed to staging/production then of course this method does not work. But the link above shows how easy it is to solve, eg by simply adding this to my migration:

$rm->deleteField('my-old-field-that-is-not-needed-any-more')

Some may call it an antipattern - but ProcessWire as a whole is a single antipattern and I really like that ? 

On 9/2/2022 at 1:36 AM, MarkE said:

Also I can attach hooks to run on installation (and uninstallation) to handle any data changes related to the migrations (e.g. say field 'name' is split into 'first_name' and 'last_name', the code can populate the new fields when the installation is run). I assume that RockMigrations can handle that too (at least for the forward install), but not quite sure how.

https://processwire.com/talk/topic/26635-add-better-configuration-for-fields-and-templates-and-make-them-version-controllable/?do=findComment&comment=220829

On 9/2/2022 at 1:36 AM, MarkE said:

I know you have commented in the past that you don't like my approach (although I'm not sure of the precise reasons**) but I think it is a valid alternative and I would like to somehow get the best of both worlds.

It's not really about liking it or not. It's just not usable for me. Why? It's simple. I like to do things in a modular, reusable way. Without any user input (beside maybe clicking an install button). I like to build modules that can be installed on many websites. And I like to deploy my websites on staging and on production automatically. Or to push changes to a project where another developer just has to do git pull and it will work. How would your module/approach do that?

On 9/2/2022 at 10:36 AM, MarkE said:

On 9/2/2022 at 1:52 AM, bernhard said:

Could you please describe exactly what you mean here?

In case you hadn't realised, there is extensive documentation of the module at https://metatunes.github.io/DbMigrate/help.html. So you don't need to install and use the module to get an understanding of what it does. Specifically as regards running code on migrations, see the following sections: https://metatunes.github.io/DbMigrate/help.html#snippets and  https://metatunes.github.io/DbMigrate/help.html#hooks.

I was confused because you wrote a specific question in my module's thread that I tried to understand, that's why I was asking for a more detailed explanation. All I got was a RTFM-like answer. But you were right, I did not know about your docs. Looks like you have put a lot of effort into them. Still I don't get how you are actually working with the module. And that was what I was asking for. To understand and maybe find a way to get the best of both worlds.

If you want to add some kind of yaml recorder or migrations GUI into RockMigrations I'm happy to pull that or to assist where I can. I'm just not adding it myself unless someone pays me to do it. Simply because I'm at the moment 90% sure that such a tool will not work for me and therefore I don't need it. But the "migrations code" inspector field is a feature coming from a push by the community, so maybe there'll be other useful things along the way as well.

[MarkE]

28 minutes ago, bernhard said:

Still I don't get how you are actually working with the module. And that was what I was asking for. To understand and maybe find a way to get the best of both worlds.

Thanks @bernhardfor your detailed response. The best of both worlds would be nice if it makes sense. First I'll do a few more improvements to my module and then try and respond to the request for a working example, before considering the way forward from there.

[gebeer]

In the past I had used the old version of RockMigrations for some simple tasks like adding fields and templates which turned out to be huge time saver. But only just recently I started to use the new version and to discover the possibilities of this tool in more depths. And what should I say, I am really amazed. Once I had grasped the core concepts and, with @bernhard's help, had learned how to use it in a more structured way and in different contexts, it turned out to to be an even bigger time saver.

I hate repetitive tasks.
Adding fields and templates and fields to templates and configuring labels / access rights etc for fields in the template context in PW is a pleasure when using the GUI. But it is still a repetitive task. With RockMigrations I can have definitions of these fields / templates and fields in template context in a single file and reuse that. In a matter of seconds I have my reusable base structure available to start off a new project.
Adding new structure through the GUI in a staging environment and then having to replicate it on the live sytem through the GUI. Repetitive task again. Pushing a migration file to the live system and running the migration. Again in a matter of seconds.

Writing migrations wherever I want is also a great feature. Be it inside site/migrate.php or inside a newly developed module, it doesn't matter. RockMigrations will find it and do its job. 

At the beginning I wasn't sure how to define different field types with all their different properties. RockMigrations comes with VSCode snippets that make things easy. Or you can create a field in the GUI and then just copy/paste the code for the migration from the GUI to your migration logic. So however you prefer to create your fields, RockMigrations has you covered.

This post may sound like an advertisement. But I just wanted to express how happy I am after having made the decision to spend some time to learn how to work with this module. That time was definitely well spent. Big thanks to Bernhard for creating this and releasing it as a free module. 

[pwired]

Hi at Gebeer, really thanks for this write up. I followed lately the posts about RockMigrations and the other Modules of Bernhard. Didn't really know how to start or apply with them. Thanks to your writing it gives me a wider perspective and motivation to jump start.

[bernhard]

Hey @gebeer thx for that great message, really happy to hear that ? 

[netcarver]

Hi @bernhard, just starting to try out your module and am having a little trouble.

I've added this file to my /site/ directory and was expecting to the a new rmdemo field and template - with the rmdemo field added to the template. Both the field and the template are created fine when I reload the modules - but the rmdemo field is not added to the rmdemo template. Am I missing something?

<?php declare(strict_types=1);
namespace ProcessWire;

/** @var RockMigrations $rm */
$rm = $this->wire->modules->get("RockMigrations");

bd('Create field + template via RM');

$rm->createField('rmdemo', 'text', [
  'label' => 'My RM Demo Field',
  'tags' => 'RMDemo',
]);

$rm->createTemplate('rmdemo', [
  'fields' => [
    'title',
    'rmdemo',
  ],
  'tags' => 'RMDemo',
]);

[netcarver]

I'm also seeing a lot of fake pages in the trash that seem to be generated by RockMigrations, despite my test file having nothing to do with either the pages or fields in the migrations.php file. For example...

[bernhard]

Hey @netcarver sorry for the troubles!!

49 minutes ago, netcarver said:

I've added this file to my /site/ directory

Is the file called migrate.php ? It looks like it should work! The only think that I don't do is strict_types=1 - maybe that causes issues? Don't know anything about that.

Do you have tracy installed? Then you can inspect the logs panel and you should see everything that RM did. And you see if it did something, because if there are no changes or your file is not watched then changes will not be applied...

31 minutes ago, netcarver said:

I'm also seeing a lot of fake pages in the trash that seem to be generated by RockMigrations, despite my test file having nothing to do with either the pages or fields in the migrations.php file. For example...

Yeah this is part of the MagicPages features introduced 2 weeks ago. It creates fake pages in the trash to make sure it can load those pages on init() and ready() and trigger their init() and ready() methods to attach hooks. I'm not happy with that solution yet and it introduced some problems on some of my projects, so I'll fix this as soon as possible! Meanwhile you could just use an older version:

https://github.com/baumrock/RockMigrations/archive/742f1fab8c727b3892835bb43d573e4f5aa7be0d.zip

[netcarver]

Hi @bernhard, yes that was my migrate.php file.  Does not seem to add anything to templates no matter what I do with it, including removing the strict declaration, though creating fields and templates seems to work (just not adding fields to templates.)

I'll try using an older version of the module.

[bernhard]

@netcarver please check v1.4.0 ? There was a wrong example in the readme having old RM1 syntax. But I've updated RM2 to use that syntax too!

[netcarver]

Thank you, Bernhard, that works fine now.

[netcarver]

Thanks for the quickstart video @bernhard, much appreciated.

[bernhard]

New video is out ? I hope it helps to get started with RockMigrations quickly! Feedback very welcome - it's really not easy to do videos about such complex topics, so if you think something is missing let me know!

Happy to hear that @netcarver ? 

[dotnetic]

RockMigrations is the best tool to do migrations with ProcessWire. It saves me so much time when creating new projects or extending older projects. Once you use it and understand it, you don't want to live without it!
Thank you very much, @bernhard.

Also a good video. I think you explained it very well, hopefully even for beginners.

What personally nags me most in every module demonstration video out there (not only from you) is how  to install a module. That is annoying to me, but might be useful for others. Gladly I can skip that part and you also added timemarks, sections to the video. Keep up the great work.

[bernhard]

2 hours ago, dotnetic said:

What personally nags me most in every module demonstration video out there (not only from you) is how  to install a module. That is annoying to me, but might be useful for others. Gladly I can skip that part and you also added timemarks, sections to the video. Keep up the great work.

Thx for the feedback @dotnetic I put that into the video for everybody to see (realise) that it's really just one click to start using migrations. No other magic that needs to be done or has been prepared only for the video. Also the migrate.php file is created on install, so I thought that is an important thing for everybody to see who is not familiar with RockMigrations ? 

2 hours ago, dotnetic said:

RockMigrations is the best tool to do migrations with ProcessWire. It saves me so much time when creating new projects or extending older projects. Once you use it and understand it, you don't want to live without it!
Thank you very much, @bernhard.

Also a good video. I think you explained it very well, hopefully even for beginners.

Thx, happy to hear that ? IMHO it is not only the best tool for migrations in ProcessWire. I've watched several videos about other migration tools in other CMSs and IMHO what we have in our PW world is much easier to use and sometimes also more powerful! The API based approach turned out to be a very reliable technique and it feels so much more natural to just add migration instructions to - for example - the migrate() of a pageclass than writing single migration files with an upgrade() and downgrade() somewhere and then end up with multiple files where you have no idea what they do (unless you inspect every single file). In RockMigrations you see what it does at first sight. And you get GIT diffs when anything changes.

@netcarver do you still want to use file-based migrations or have you already changed your mind? I mean... you can also do that with RockMigrations - it's just not what I'm doing but that does not mean it's not possible or bad ? Maybe there are also good reasons for such an approach, so I'm happy to get input.

[netcarver]

Hi Bernhard, no, all good at present.

[zoeck]

12 hours ago, bernhard said:

New video is out ?

Woah... i love the MagicPages Feature ? (and also all the rest ?)
I'm still using v1 of RockMigrations, but i think it's time to switch to the new version ? 

Thanks for the great video bernhard!