Flowti Core - RockMigrations Companion

[LuisM]

A companion Module for RockMigrations to gather and execute Migrations in one place. !Alpha RELEASE!

Hi there,

I just wanted to share my take on Migrations and how I am using RockMigrations to make my deployments somehow manageable. 

First of all, a big shoutout to @bernhard for his efforts and accomplishments for our beloved CMS. 
We recently had a video-call and talked about our daily problems and how to handle them, with a focus on Migrations. As of now you might think 'Gosh, not another Migrationsthingy' but wait. 

Wouldnt it be nice to have a nice overview or some kind of list regarding Migrations and stuff and one place to manage them?

Speaking for my own, yes I would like to have something like this. So yeah, here we are with a Companion Module for RockMigrations. 

 

 

Product Requirements

• I want to manage Migrations on the Application level (done)
• I want to have one place in my FileSystem to place my Migrations (done)
• I want to see in which state my Migrations are (done)
• I want to have an execution order, execute migrations one by one (done)
• I want to trigger Migrations via CLI (open)
• I want to group multiple Migrations into One Migration to create Build Versions (open)
• I want to Rollback Migrations (open)
• I want to create a deliverable build out of migrations (open)
• I want to track changes to templates and fields and create migrations accordingly (open)
• I want to manage Migrations on the Module level (open)

Module Requirements

• ProcessWire 3.0.178
• RockMigrations latest
• PHP 7.4
• Composer

Current release: v1.0.2Alpha

How does it work?

On installation the Module will create a new Database Table, here we will hold information about migrations states.

Your Migrations are empty on first installation, so just click on 

You can now choose which Type of Migration you want to create, be it Templates, Fields or Pages and a according Action. 
Im still working on this function, so yeah its a lil bit confusing right now. 

The philosophy here is, every Type of Migration could be easily identified and we can Migrate on a very sepcific and granular base. As you can see in my Screenshot, I am using migrations to build an entire App from Migrations.

After creating the new Migration File, switch over to your IDE and find a timestamped .php file inside modules/FlowtiCore/migrations. 
This file just returns a simple Array.

This is all the Info RockMigrations needs to do his thing while we keep track of the execution. Easy.

Values Arrays have to follow PW and RockMigration field naming Conventions. 

To make a Migration Executable set 'boilerplate' to false.

After creation of your first Migrations your overview should be filled so lets go. We have to migrate our files in a specific order just to make sure we can create page_references and Parent/Child connections. To achieve this, Migrations are timestamped on creation. The older the higher the priority. To enforce execution order, every migration needs to have his predecessor executed and installed.

How could this help my workflow?

My workflow is based on Git, Webhooks and Git Events. So whenever I merge 'staging' into 'master', a build will be created and a deliverable should be pushed to a Server via SSH. 

The Problem with ProcessWire is the lack of support for such Workflows and Toolchains due to its User-Friendly Admin Backend which is fine for a simple website but not suitable long-term if working in a multi-tenant environment or with more developers in a dev-staging-test-production setup.

My Goal is to provide methods and ideas to support such workflows but also support a User-Friendly Interface to work with migrations.

I really hope it could be of use for someone.

Installation

I will add the Module to the Directoy once it reached a stable state.

But you can get current version at GitHub

https://github.com/Luis85/FlowtiCore

Just clone it to your Modules Directory.

The Module will create a new Database-Table on creation. The Default Name will be 'flowti_core_migrations'. To change this just edit 

const DATABASE_TABLE = 'flowti_core_migrations';

Inside the Module Class. 

Thats it, from there on just create new Migration Files, edit them and execute them.

[wbmnfktr]

To be honest... what you @LuisM and @bernhard built... it's somehow awesome but either way somehow insane.

I love it... even though I almost will never use it by myself... as it's either too complex or way out of my comfort zone. (for now)

Yet... I really appreciate what you @bernhard do for the community with all your modules and ideas and you @LuisM who builds things on top of it.

Love it!

[LuisM]

thanks for the warm words webmanufakturer @wbmnfktr.

I am happy to walk you trough the concepts of migrations if you want. The most complexity should be abstracted away with RockMigrations and a companion.

As a note regarding my module, the FlowtiMigrationsService class has a function called addDefaultValues, I use this method to keep my migrations a lil bit lighter.

Unfortunatly, I have to maintain 2 Versions of the module right now. The public version and our internal module.

Therefore I forgot to empty the defaultValues Function.

I try to push a cleaned up Version later today with a configurable AddDefaults method.

[LuisM]

Pushed to Master

• renamed FlowtiMigration::createMigrationFile() to ::createMigrationBoilerplateFile()
• renamed FlowtiMigrationService::addDefaultMigrationValues() to ::addMigrationValues()
• Moved default Value creation in his own function, added default values array with all options
• cleanup

You cann now define your defaults/globals for all Templates and Fields. To set a default Value just remove '//' for your particular option inside the src/FlowtiMigrationService class.

I pushed my current default values, so please change them as you need them ?

[bernhard]

Hi @LuisM

thx for the great writeup - it's always hard to understand complex topics! To be honest I'm not sure if I like what I see ? On the one hand sometimes I wished I had some kind of overview/list of migrations and some place to manage them, but on the other hand I'm using migrations in a totally different way and your approach feels like a step back in many ways. No offense here - both approaches have pros and cons and it might simply be a matter of preference ? 

The problems with that approach are the following (for me):

• One central place for migrations
  
  That might sound good and it might be superior from a technical aspect, but for me and my daily work it's not. I'm building my sites with many different modules and my main goal is productivity and quality. Almost anything that I do is done via modules nowadays. These modules do often need to create the necessary fields and templates, and they do that using RockMigrations. It's great. I get a module that is reusable across projects and it's quite easy to develop and to create all the necessary database stuff. And it's easy to push new versions to production and after a modules refresh I have all that I need.
  
  Having a central place for migrations locks all the work that I do in one single (and not reusable) project. Or am I misunderstanding your concept?
  
  So my solution would be: Create different modules that do different things (eg a newsletter module, a sitemap module, a slider module) and all those modules ship with their own migrations. They are separate pieces and can be used across several projects. Then on the main project I'd have another module (sometimes I simply call that Site.module.php) and that one has migrations to install other modules or do stuff that does not fit into a reusable module. Such a migration could look like this:
   

  $rm->installModule('Slider');
  $rm->installModule('Sitemap');
  // create home page
  // set page title
  // create imprint page
  // etc etc

  
  I know that this approach has other drawbacks, but for now it works quite good for me ? 
   
• Template/Field/Page migrations
  
  I think I also don't like the approach of splitting every migration into a dedicated field/template/page migration. Maybe I'm misunderstanding you again, but usually such migrations are related. Eg you want to add a field to a template so you need to create that field and then you need to add that field to your template. Having those changes in seperate files leads to a mess of migration files that are very hard to understand afterwards. You lose the big picture imho.
  
  My approach is to place all that in the migrate() method and you'll end up with a static "snapshot" of your system that you'll easily and instantly understand when you look at it. That get's even better when you use GIT:
  
  
  The drawback is of course that it get's harder to revert such changes. Backup/Restore is one technique or custom downgrade migrations could be another concept. What I sometimes do is adding a "cleanup()" method on top of the migrations that removes fields that I do not need any more. So if I needed to remove that reminder field, I'd do the following:
  
  
  
  Again: Reverting would be a challenge, but I've never needed any reverts until now and backup/restore has served me well enough ? 

Hope the post was helpful nevertheless ? 

[bernhard]

PS: Thanks for your interest and efforts in migrations - always very welcome to see someone is working on solutions ? ? 

[LuisM]

Bernhard,

thanks for your detailed and well tought answer, really appreciate it. 

I definitly see where you are coming from. 
We are working both on very different topics, you are working on a multitude of projects and you need to keep track of your modules and the various different combinations and connections between them. I am glad I dont have to deal with this ?

My usecase is a bit different as I am working on a single Installation with a well defined set of Modules and the same functions for every single user/customer. Your approach to deal with Migrations on a Module Base makes sense, yes. Thing is, I really dont like to mix Business Logic, Domain specific Logic and Application Logic hence my file based approach to solve my first need. 
My second need would be to put Modules under Migrations Control like you do. 

My primary thought was to create a /site/migrations/ folder and let the User decide which Module he wants to set under Migrations Control and act accordingly.
So moving the Module bound migrations folder into the site scope. I dunno yet ? but I will definitly support a usecase as you have in the future, just because I would need that feature, too ?

Regarding the granularity of Migrations, yes, it could end in a mess. My reasoning to seperate the different types of Migrations is to force the developer to atleast think about execution order and implications in his database change. You are forced to discipline yourself so to say, to avoid a mess ? #me #myself and #i

[bernhard]

Yeah. That makes sense ?

Just thought maybe your approach and mine could somehow be merged? Eg maybe migrations could not only live in /site/migrations but also in /site/modules/mymodule1/migrations and /mymodule2/migrations ? So each module could have a separate view of what you already have for the central migrations?

Just brainstorming ? 

[LuisM]

I had your workflow in mind when creating the Module ? But needed a baseline to get it going.

I will update the Product Requirements, just hadnt tought enough yet how to merge the different approaches. 

My inital thought was to let the user define Modules he want to manage with the module, build an array of Modules and Folders according to the user configuration and scan affected folders.
Which will result in a structure like

• /site/migrations 
• /site/modules/XYZ/migrations

On a sidenote, I also want to add Template and Field changes done via the ProcessWire Backend under Migration Control just to support a easy "click-and-collect" Workflow. 
For example you are building some Templates via UI to test things out on your local machine, Migrations will be added automagically and after a Push to your GitRepo you will have your executable Migrations in place.

[bernhard]

10 minutes ago, LuisM said:

On a sidenote, I also want to add Template and Field changes done via the ProcessWire Backend under Migration Control just to support a easy "click-and-collect" Workflow. 
For example you are building some Templates via UI to test things out on your local machine, Migrations will be added automagically and after a Push to your GitRepo you will have your executable Migrations in place.

That would be super cool stuff ?

[wbmnfktr]

Hey @LuisM... just a short feedback on this.

On 6/29/2021 at 9:15 AM, LuisM said:

I am happy to walk you trough the concepts of migrations if you want. The most complexity should be abstracted away with RockMigrations and a companion.

I'm really happy you would take your time to guide me and probably anyone else trough this setup and what's possible with it. I already talked to @bernhard in the past several times in regards to his module as I was super interested how all that works. I'm still impressed what he does with that setup.

I really really love those kinds of setups, workflows and automations... yet... the initial setup of RockMigration is already way larger and more complex than any of my projects.

My projects (look at muskaat.de) are most of the time super simple and straight forward. They do what they need to do... sometimes more complex, sometimes only a few more files, and most of the time only a DEV and LIVE stage. Nothing I'd need such a setup for. Even though I really am a Spielkind (someone that loves to play around with all possible options).

Still... I know and understand the benefits of those setups with RockMigrations - and your addition to it - and what they could and can do. As I'm most of the time the one and only developer for now... Git is perfectly fine for any workflow I can imagine.

I'm missing two or three developers on my side and maybe a few more stages, like dev / qa / testing / preprod / live, within my projects to really benefit from your and @bernhard's modules.

 

Yet... I really appreciate all the things you both and several others here build to make ProcessWire even more awesome for whoever.