Jump to content

Flowti Core - RockMigrations Companion


LuisM
 Share

Recommended Posts

A companion Module for RockMigrations to gather and execute Migrations in one place. !Alpha RELEASE!

Hi there,

I just wanted to share my take on Migrations and how I am using RockMigrations to make my deployments somehow manageable. 

First of all, a big shoutout to @bernhard for his efforts and accomplishments for our beloved CMS. 
We recently had a video-call and talked about our daily problems and how to handle them, with a focus on Migrations. As of now you might think 'Gosh, not another Migrationsthingy' but wait. 

Wouldnt it be nice to have a nice overview or some kind of list regarding Migrations and stuff and one place to manage them?

Speaking for my own, yes I would like to have something like this. So yeah, here we are with a Companion Module for RockMigrations. 

 

 

Product Requirements

  • I want to manage Migrations on the Application level (done)
  • I want to have one place in my FileSystem to place my Migrations (done)
  • I want to see in which state my Migrations are (done)
  • I want to have an execution order, execute migrations one by one (done)
  • I want to trigger Migrations via CLI (open)
  • I want to group multiple Migrations into One Migration to create Build Versions (open)
  • I want to Rollback Migrations (open)
  • I want to create a deliverable build out of migrations (open)
  • I want to track changes to templates and fields and create migrations accordingly (open)
  • I want to manage Migrations on the Module level (open)

Module Requirements

  • ProcessWire 3.0.178
  • RockMigrations latest
  • PHP 7.4
  • Composer

Current release: v1.0.2Alpha

How does it work?

On installation the Module will create a new Database Table, here we will hold information about migrations states.

Your Migrations are empty on first installation, so just click on 

add-migration-file-button.thumb.jpg.bd1e8fde90cf5616133a90b3ac8a6590.jpg

You can now choose which Type of Migration you want to create, be it Templates, Fields or Pages and a according Action. 
Im still working on this function, so yeah its a lil bit confusing right now. 

The philosophy here is, every Type of Migration could be easily identified and we can Migrate on a very sepcific and granular base. As you can see in my Screenshot, I am using migrations to build an entire App from Migrations.

After creating the new Migration File, switch over to your IDE and find a timestamped .php file inside modules/FlowtiCore/migrations. 
This file just returns a simple Array.

migration-file-preview.thumb.jpg.a268e9b60a76898e92b44fa44bea3dee.jpg

This is all the Info RockMigrations needs to do his thing while we keep track of the execution. Easy.

Values Arrays have to follow PW and RockMigration field naming Conventions

To make a Migration Executable set 'boilerplate' to false.

After creation of your first Migrations your overview should be filled so lets go. We have to migrate our files in a specific order just to make sure we can create page_references and Parent/Child connections. To achieve this, Migrations are timestamped on creation. The older the higher the priority. To enforce execution order, every migration needs to have his predecessor executed and installed.

core-overview-execution-order.thumb.jpg.b9852366cee2f482ec753a942d520613.jpg

How could this help my workflow?

My workflow is based on Git, Webhooks and Git Events. So whenever I merge 'staging' into 'master', a build will be created and a deliverable should be pushed to a Server via SSH. 

The Problem with ProcessWire is the lack of support for such Workflows and Toolchains due to its User-Friendly Admin Backend which is fine for a simple website but not suitable long-term if working in a multi-tenant environment or with more developers in a dev-staging-test-production setup.

My Goal is to provide methods and ideas to support such workflows but also support a User-Friendly Interface to work with migrations.

I really hope it could be of use for someone.

Installation

I will add the Module to the Directoy once it reached a stable state.

But you can get current version at GitHub

https://github.com/Luis85/FlowtiCore

Just clone it to your Modules Directory.

The Module will create a new Database-Table on creation. The Default Name will be 'flowti_core_migrations'. To change this just edit 

const DATABASE_TABLE = 'flowti_core_migrations';

Inside the Module Class. 

Thats it, from there on just create new Migration Files, edit them and execute them.

core-overview.jpg

core-overview-createmigration.jpg

database-migrations.jpg

  • Like 6
  • Thanks 1
Link to comment
Share on other sites

To be honest... what you @LuisM and @bernhard built... it's somehow awesome but either way somehow insane.

I love it... even though I almost will never use it by myself... as it's either too complex or way out of my comfort zone. (for now)

Yet... I really appreciate what you @bernhard do for the community with all your modules and ideas and you @LuisM who builds things on top of it.

Love it!

  • Like 3
  • Thanks 1
Link to comment
Share on other sites

thanks for the warm words webmanufakturer @wbmnfktr.

I am happy to walk you trough the concepts of migrations if you want. The most complexity should be abstracted away with RockMigrations and a companion.

As a note regarding my module, the FlowtiMigrationsService class has a function called addDefaultValues, I use this method to keep my migrations a lil bit lighter.

Unfortunatly, I have to maintain 2 Versions of the module right now. The public version and our internal module.

Therefore I forgot to empty the defaultValues Function.

I try to push a cleaned up Version later today with a configurable AddDefaults method.

  • Like 2
Link to comment
Share on other sites

Pushed to Master

  • renamed FlowtiMigration::createMigrationFile() to ::createMigrationBoilerplateFile()
  • renamed FlowtiMigrationService::addDefaultMigrationValues() to ::addMigrationValues()
  • Moved default Value creation in his own function, added default values array with all options
  • cleanup

You cann now define your defaults/globals for all Templates and Fields. To set a default Value just remove '//' for your particular option inside the src/FlowtiMigrationService class.

defaultvalues_serviceclass.jpg.ef80fe9bf9fbc117cf82a7ecf508d9e6.jpg

I pushed my current default values, so please change them as you need them 🙂

  • Like 1
Link to comment
Share on other sites

Hi @LuisM

thx for the great writeup - it's always hard to understand complex topics! To be honest I'm not sure if I like what I see 😄 On the one hand sometimes I wished I had some kind of overview/list of migrations and some place to manage them, but on the other hand I'm using migrations in a totally different way and your approach feels like a step back in many ways. No offense here - both approaches have pros and cons and it might simply be a matter of preference 🙂 

The problems with that approach are the following (for me):

  • One central place for migrations

    That might sound good and it might be superior from a technical aspect, but for me and my daily work it's not. I'm building my sites with many different modules and my main goal is productivity and quality. Almost anything that I do is done via modules nowadays. These modules do often need to create the necessary fields and templates, and they do that using RockMigrations. It's great. I get a module that is reusable across projects and it's quite easy to develop and to create all the necessary database stuff. And it's easy to push new versions to production and after a modules refresh I have all that I need.

    Having a central place for migrations locks all the work that I do in one single (and not reusable) project. Or am I misunderstanding your concept?

    So my solution would be: Create different modules that do different things (eg a newsletter module, a sitemap module, a slider module) and all those modules ship with their own migrations. They are separate pieces and can be used across several projects. Then on the main project I'd have another module (sometimes I simply call that Site.module.php) and that one has migrations to install other modules or do stuff that does not fit into a reusable module. Such a migration could look like this:
     
    $rm->installModule('Slider');
    $rm->installModule('Sitemap');
    // create home page
    // set page title
    // create imprint page
    // etc etc

    I know that this approach has other drawbacks, but for now it works quite good for me 🙂 
     
  • Template/Field/Page migrations

    I think I also don't like the approach of splitting every migration into a dedicated field/template/page migration. Maybe I'm misunderstanding you again, but usually such migrations are related. Eg you want to add a field to a template so you need to create that field and then you need to add that field to your template. Having those changes in seperate files leads to a mess of migration files that are very hard to understand afterwards. You lose the big picture imho.

    My approach is to place all that in the migrate() method and you'll end up with a static "snapshot" of your system that you'll easily and instantly understand when you look at it. That get's even better when you use GIT:
    Z1otlCt.png

    The drawback is of course that it get's harder to revert such changes. Backup/Restore is one technique or custom downgrade migrations could be another concept. What I sometimes do is adding a "cleanup()" method on top of the migrations that removes fields that I do not need any more. So if I needed to remove that reminder field, I'd do the following:

    1EIzKjJ.png

    Again: Reverting would be a challenge, but I've never needed any reverts until now and backup/restore has served me well enough 😄 

Hope the post was helpful nevertheless 🙂 

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

Bernhard,

thanks for your detailed and well tought answer, really appreciate it. 

I definitly see where you are coming from. 
We are working both on very different topics, you are working on a multitude of projects and you need to keep track of your modules and the various different combinations and connections between them. I am glad I dont have to deal with this 😄

My usecase is a bit different as I am working on a single Installation with a well defined set of Modules and the same functions for every single user/customer. Your approach to deal with Migrations on a Module Base makes sense, yes. Thing is, I really dont like to mix Business Logic, Domain specific Logic and Application Logic hence my file based approach to solve my first need. 
My second need would be to put Modules under Migrations Control like you do. 

My primary thought was to create a /site/migrations/ folder and let the User decide which Module he wants to set under Migrations Control and act accordingly.
So moving the Module bound migrations folder into the site scope. I dunno yet 🙂 but I will definitly support a usecase as you have in the future, just because I would need that feature, too 🙂

Regarding the granularity of Migrations, yes, it could end in a mess. My reasoning to seperate the different types of Migrations is to force the developer to atleast think about execution order and implications in his database change. You are forced to discipline yourself so to say, to avoid a mess 😄 #me #myself and #i

  • Like 3
Link to comment
Share on other sites

Yeah. That makes sense 🙂

Just thought maybe your approach and mine could somehow be merged? Eg maybe migrations could not only live in /site/migrations but also in /site/modules/mymodule1/migrations and /mymodule2/migrations ? So each module could have a separate view of what you already have for the central migrations?

Just brainstorming 😉 

  • Like 1
Link to comment
Share on other sites

I had your workflow in mind when creating the Module 🙂 But needed a baseline to get it going.

I will update the Product Requirements, just hadnt tought enough yet how to merge the different approaches. 

My inital thought was to let the user define Modules he want to manage with the module, build an array of Modules and Folders according to the user configuration and scan affected folders.
Which will result in a structure like

  • /site/migrations 
  • /site/modules/XYZ/migrations

On a sidenote, I also want to add Template and Field changes done via the ProcessWire Backend under Migration Control just to support a easy "click-and-collect" Workflow. 
For example you are building some Templates via UI to test things out on your local machine, Migrations will be added automagically and after a Push to your GitRepo you will have your executable Migrations in place.

  • Like 4
Link to comment
Share on other sites

10 minutes ago, LuisM said:

On a sidenote, I also want to add Template and Field changes done via the ProcessWire Backend under Migration Control just to support a easy "click-and-collect" Workflow. 
For example you are building some Templates via UI to test things out on your local machine, Migrations will be added automagically and after a Push to your GitRepo you will have your executable Migrations in place.

That would be super cool stuff 😎

  • Like 1
Link to comment
Share on other sites

Hey @LuisM... just a short feedback on this.

On 6/29/2021 at 9:15 AM, LuisM said:

I am happy to walk you trough the concepts of migrations if you want. The most complexity should be abstracted away with RockMigrations and a companion.

I'm really happy you would take your time to guide me and probably anyone else trough this setup and what's possible with it. I already talked to @bernhard in the past several times in regards to his module as I was super interested how all that works. I'm still impressed what he does with that setup.

I really really love those kinds of setups, workflows and automations... yet... the initial setup of RockMigration is already way larger and more complex than any of my projects.

My projects (look at muskaat.de) are most of the time super simple and straight forward. They do what they need to do... sometimes more complex, sometimes only a few more files, and most of the time only a DEV and LIVE stage. Nothing I'd need such a setup for. Even though I really am a Spielkind (someone that loves to play around with all possible options).

Still... I know and understand the benefits of those setups with RockMigrations - and your addition to it - and what they could and can do. As I'm most of the time the one and only developer for now... Git is perfectly fine for any workflow I can imagine.

I'm missing two or three developers on my side and maybe a few more stages, like dev / qa / testing / preprod / live, within my projects to really benefit from your and @bernhard's modules.

 

Yet... I really appreciate all the things you both and several others here build to make ProcessWire even more awesome for whoever.

  • Like 3
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Similar Content

    • By DooM
      Hello guys,
      I'm trying to figure out how to sync fields and templates between staging and production environments.
      I've found Migrations module by Lostkobrakai, but with use of it all the fields and templates must be created by API, which is kind of uncomfortable.
      I also tried ProcessDatabaseBackups module which can export only certain tables, but I don't think it's the best practice to do that.
      How do you guys solve this problem? It's very annoying to setup everything three times (dev, staging, production).
      Thanks a lot :)
    • By bora
      Hello everyone,
      Last night as a "saturday night shitty weather stay at home" project I attempted to migrate a project that was on PW 2.7.x to PW 3.x 
      I use a lot of partials on the project so I have around 200 php files that needs to be namespaced. Compiler was giving me trouble with "Call undefined function" errors.
      Being the lazy developer I attempted to wrote a script after getting bored over pasting  the namespace Processwire; line into around 10 files.
      Below you can find the script, that is very basically adds <?php namespace Processwire;?> as the first line of every .php and .module file in the given directory and shows you the results. Of course it checks for namespace Processwire first
      By default it assumes ./site/templates folder but I tried it with a module which is giving errors due to namespaces and worked fine.
      Be careful and remember to take backups first
      https://gist.github.com/borantula/e41c4b6ba36f78b1110d400a16754691
×
×
  • Create New...