Migrations

[LostKobrakai]

This module is deprecated in favor of RockMigrations. It'll continue to work and I might fix some smaller incompatibilities if they're reported, but no major development will happen on this anymore.

There where various threads about a how to reasonably handle multiple dev/staging and live environments with ProcessWire and at best handle it as automatically as possible. A git based workflow makes it easy to handle files, but the pain point of migrating db changes has even lead to multiple requests of not handling template/field storage in the db at all.

I've gone ahead and used for my own projects the concept of database migrations, which most frameworks are using to handle database changes – but the ProcessWire flavored way.

    ___       ___       ___       ___       ___       ___       ___
   /\__\     /\  \     /\  \     /\  \     /\  \     /\  \     /\  \
  /::L_L_   _\:\  \   /::\  \   /::\  \   /::\  \    \:\  \   /::\  \
 /:/L:\__\ /\/::\__\ /:/\:\__\ /::\:\__\ /::\:\__\   /::\__\ /::\:\__\
 \/_/:/  / \::/\/__/ \:\:\/__/ \;:::/  / \/\::/  /  /:/\/__/ \:\:\/  /
   /:/  /   \:\__\    \::/  /   |:\/__/    /:/  /   \/__/     \:\/  /
   \/__/     \/__/     \/__/     \|__|     \/__/               \/__/

Beta

This module does help you managing migration files, where any database changes can be stored in php files using just the simple ProcessWire api at your disposal. It's not as nice as using the admin UI, but certainly better than trying to migrate changes manually and possibly weeks after adding the changes. Also there's always the option to create helper modules, which can export changes made in the admin UI to something usable in those migration files. For example I'm using an internal process module, which does let me export changes to template access rules as json strings, which I then use in my migrations to actually apply the changes.

Now on to the meat – How to use the module:

• Read the following guide on creating own migrations
• Maybe use the CLI tool to speed up your workflow (and possibly automate stuff)

It is generally recommended, but not enforced, that migrations are run/rolled back in order. When doing migrations or rollbacks, without specifying a migration, this module will stick to the order.

Creating Migrations

Your migrations will probably hold lot's of code, which does delete data. By now this module does not have any security measurements to prevent that. Be sure to test your migrations locally and possibly keep a database backup before running them.

There are currently four types of migrations:

default (Migration)

Default migrations are the most free form migrations. There's just a description and two functions – update() and downgrade(). What you're doing in those functions is totally up to you, but it's recommended to try the best to keep changes as reversible as possible. Meaning that running update() and downgrade() once should have as less effect on the installation as possible. The ProcessWire API is available exactly like in modules using the $this->pages, $this->config, … syntax.

FieldMigration

TemplateMigration

ModuleMigration

All of those are there to make your life easier. They all have different but similar functions – which you can find in migrations created by this module – which ease the creation and removal of fields, templates or modules. All the boilerplate is handled by the base classes these migrations do extend, so you don't even need to think about update() and downgrade(). You can rather just describe the item you want to handle and the creation / removal process is taken care of. 

These are by now not highly tested, so please again be sure to test migrations before running them on important content.

Command-Line Interface

The module does include a cli interface, which does allow the migrations to be run automatically by CI or deployment scripts or just by you if you like cli's. The cli script is located in the bin directory inside the module's folder. It does however require a composer package to work, which you can simply add by running composer require league/climate in your site directory (or the root directory for pw 3.0). Make sure to require composers autoload.php in your config.php for 2.x installations. 

The CLI does have a quite handy help page, which you get by running php migrate -h so I'm just adding the important bits of that here:

> php migrate -h

[…]

Usage: migrate [-h, --help] [-i info, --info info] [-m migrate, --migrate migrate] [-n new, --new new] 
  [-nf newField, --newField newField] [-nm newModule, --newModule newModule] 
  [-nt newTemplate, --newTemplate newTemplate] [-r rollback, --rollback rollback]

Optional Arguments:
	-m migrate, --migrate migrate
		Run a specific migration or all new* ones if none given.
		* From latest migrated to newest.
	-r rollback, --rollback rollback
		Undo a specific migration or the latest one if none given.
	-n new, --new new
		Bootstrap a new migrations file. Optionally you can already supply a description.
	-nt newTemplate, --newTemplate newTemplate
		Bootstrap a new template migrations file. Optionally you can already supply a description.
	-nm newModule, --newModule newModule
		Bootstrap a new module migrations file. Optionally you can already supply a description.
	-nf newField, --newField newField
		Bootstrap a new field migrations file. Optionally you can already supply a description.
	-i info, --info info
		Get detailed info about a migration.
	-h, --help
		Show all commands of the cli tool.

Link the migrations cli update save to ProcessWire's root: https://processwire.com/talk/topic/13045-migrations/?p=118329

Helper Functions

There are already a handful of helper function included in the Migration base class, which tackle things I found to need way to much boilerplate for kinda simple changes, but you can also add own custom helper functions via hooks.

/**
 * This does use @diogo's while loop technique to loop over all pages 
 * without getting memory exhaustion. 
 */
$this->eachPageUncache("template=toBeHidden", function($p){
  $p->setAndSave('status', Page::statusHidden);
}); 

/**
 * $template, $field, $reference = null, $after = true
 * The below function reads like this:
 * In the template … add the field … relative to the field … in the position after/before
 */
$this->insertIntoTemplate('basic-page', 'images', 'body', false); 

/**
 * Edit field settings in context of a template
 */
$this->editInTemplateContext('basic-page', 'title', function($f, $template){
  $f->label = 'Headline';
}); 

And a simple example of adding a custom helper as a hook.

// in ready.php
$wire->addHook('Migration::renameHome', function(HookEvent $event){
	$name = $event->arguments(0);
	wire('pages')->get('/')->setAndSave('title', $name);
});

// in the migration
$this->renameHome('Root'); 

Snippets

Still not sure how all this works in practice? Or you want to share a maybe more complex migration? Just head over to the Snippets Repo at Github. 

https://github.com/LostKobrakai/MigrationSnippets

There are also less specific examples in the modules repository:

https://github.com/LostKobrakai/Migrations/tree/master/migrations

Appendix

As long as the module is in Beta the helper functions in the Migration.php might be object to change/removal, so be aware of that.

Download

http://mods.pw/Bm

https://github.com/LostKobrakai/Migrations

[netcarver]

Looks great, Benjamin, thank you! I have not tried this yet, but look forward to doing so.

Could your module check if ProcessDatabaseBackups is installed? If it is perhaps it could take care of either backing up the DB or at least reminding the user to do so, before it performs a migration.

[LostKobrakai]

Sure. I just wanted to get it out, but this can certainly being added. We're using Cronjob DB Backup on our project so it's not been so much a necessity for me. Also it wouldn't even need to check for anything as the backup functionality is part of the core. Only the admin UI must be installed.

[Jonathan Lahijani]

Fantastic.  Been waiting very long for something that takes me back to the days of Rails migrations!

I will be doing a screencast on this in my WP vs. PW series.  Just in time!

[LostKobrakai]

I've just commited a few smaller updates (most on Jonathan's requests)

• Migration descriptions are now escaped, so that characters like double quotes or backslashes won't break any newly created migrations. Also the backend UI does now feature a textarea instead of a text input to allow for multiline description entry. In the cli one would have to add "\n" linebreaks manually.
• The module does now try to add the /site/migrations/ path automatically, when it's called for the first time (e.g. by accessing the admin UI page)
• I've added a basic onboarding page for the admin UI, when there are no migrations, which does now feature a "Create New" button to get started.

Jonathan did also request the addition of importer migrations for modules like FormBuilder or HannaCode. I'm not really keen on adding those and especially maintaining them. I'd rather suggest creating a GitHub repo, where the community can add migration snippets via pull-request. For example using the FormBuilder importer is just a single line of code, whereas the HannaCode importer would need to be copied nearly completely, because it's so tightly coupled with it's backend form. This repo option would be an easy way to just share the bits of boilerplate which is sometimes needed to do things via the api.

Additionally I'm going to add some example above on how to add own helper methods to the Migration class with hooks, so everybody can add his own helper functions.

Edit:

I've added the mentioned docs about helper functions.

[LostKobrakai]

For those who would rather like the cli tool to live in another directory (e.g. root in my case) you can use this in a file named 'migrate':

<?php
// Full or relative path to the module's cli
include_once __DIR__ . '/site/modules/Migrations/bin/migrate';

[bernhard]

https://github.com/LostKobrakai/Migrations/blob/master/Migration.php#L11-L30

This does use @diogo's while loop technique to loop over all pages 

could you please provide a link or more information on that

[kongondo]

https://github.com/LostKobrakai/Migrations/blob/master/Migration.php#L11-L30

could you please provide a link or more information on that

I believe this is the one: https://processwire.com/talk/topic/2038-distinct-selector-in-processwire/?p=43720

[bernhard]

thank you kongondo, now it makes sense! there's very much pw und php wisdom in this one method. very nice

[LostKobrakai]

I just added a small code snippet so that custom global fields are also added to templates when using a TemplateMigration.

[teppo]

"There's already a file existing for the current time." Apparently I can only create a migration file once a minute. Is there a reason for this or am I just being too hasty?

[LostKobrakai]

I've created the name format this way, when I started creating this module for my own purposes and I never had it matter except when testing things right before releasing the module here. But I'm considering to add seconds to the timestamp as well, especially as those new template and field migrations can really speed up the creation of those migrations.

[LostKobrakai]

I've just added the module to the modules directory. As soon as it's reviewed it'll be available here: mods.pw/Bm (is it a coincidence, that the shortcode is my name's initial letters)

[Miguel Scaramozzino]

This is amazing. Migrations are the only feature I really missed from my FuelPHP days, a framework I stopped using after switching to PW. Cannot wait to try this out, thank you very much!

[LostKobrakai]

Glad to hear that. If some of you guys are using it I'd really like to hear your thoughts or issues.

[pmichaelis]

Love it! Great work.

[LostKobrakai]

On 16.4.2016 at 11:46 AM, LostKobrakai said:

Jonathan did also request the addition of importer migrations for modules like FormBuilder or HannaCode. I'm not really keen on adding those and especially maintaining them. I'd rather suggest creating a GitHub repo, where the community can add migration snippets via pull-request. For example using the FormBuilder importer is just a single line of code, whereas the HannaCode importer would need to be copied nearly completely, because it's so tightly coupled with it's backend form. This repo option would be an easy way to just share the bits of boilerplate which is sometimes needed to do things via the api.

I've just gone ahead and added such a repo to GitHub: https://github.com/LostKobrakai/MigrationSnippets. Maybe this can also serve as a resource for people, who are not sure they understand what this module does provide. I've already added 2 example cases. I think over time I'll add some more, but everyone is welcome to add their files as well.

[LostKobrakai]

I've just added a Snippet about reversing the functionality of the build-in (or custom) MigrationTypes. 

https://github.com/LostKobrakai/MigrationSnippets/blob/master/Reverse_Template_Migration_Type.php

[Michael Murphy]

Thanks for your work on this, it’s a feature many of us have been after for a while and is much appreciated.

I have been playing around with the module and I think I understand how it works, but I would appreciate some further clarification on how I can use it. 

Modifying or adding templates + fields
My most common use case is migrating field and template changes from my local development to a production server.  
What I currently do is
1. check in my template php file changes
2. deploy files to production
3. backup production database.
4. apply the template / field changes on the production site by exporting the json files from /setup/(template|field)/export/ and importing them via /setup/(template|field)/import/ within the PW admin.

With this module, can I now copy these json files into a migration file and deploy them with the rest of my files, and then run the migrations module on the production site?

Is there any way to “record” or log all template and field changes as you are developing and then import them into a migrations script?

Other use cases
Some other scenarios that I would like to use this module for…
Changing existing data
Like changing all users with role “editor” to use a new admin theme.
Migrating from one field to another
Like migrating pages from a PageTable field to a RepeaterMatrix field (something I am planning to do!)

Collaboration workflow
I am very interested to hear how this could be used with a team of developers working on the same PW project. How would you keep all the developers and designers using the latest version, notifying them when they need to run a migrations script etc. Something like this…
1. Store project files in version control including migration script
2. Push files and deploy
3. Other team members pull latest version - they are somehow notfied of a new migrations script to run (in PW admin maybe)
3. Backup db
4. Run migrations
5. Everyone hugs each other to celebrate all the extra time they have saved while doing migrations.

Upgrading site profiles  
Could this be used to manage site profiles and keep them updated? For example, if Ryan makes some field / template changes to the blog profile, could this module be used to update existing users who already have the blog profile installed? It would be nice to work out a way to have an upgrade path for site profile development.

Thanks!

[LostKobrakai]

I'm on mobile, but I try to answer those questions as best as possible.

For migrating field and template changes I'm currently not specifically supporting those json exports. The first reason is quite simple in that the json export didn't support all field features at the time I created the module. The other one is the fact, that I'm not really convinced that adding migrations after the fact is really a good workflow improvement. I'm doing all my local changes via migrations as well. This "ensures" their correctness, is after a few weeks of working with migrations not considerably slower than doing it in the UI and it's really recording your changes 100%. Another one just poping into my head is that those json strings are only exporting a current state. If you want the option to rollback changes there's also the need to have some way to undo changes, which those exports do not provide. I'd suggest you to give the template/field migrations a try/look for creation/deletion of them and also some of my helper functions, which help adding fields to templates and such things. I'm also open for suggestions of other helper functions if they hide a lot of unnecessary boilerplate code.

Just to make that clear. The module is certainly not meant to record any UI changes for the user.

Those other things you mentioned you want to be able to change are all certainly doable. You're free to do whatever you need in the default migrations. 

The collaboration issues is nothing I had to deal with very much by now, but I'm getting a collaborator in a few weeks, so there might be updates coming. The biggest issue is the fact that db's are shitty to keep in sync with vcs branches. If you can create a baseline db state you can simply tell collaborators to init that baseline db and run all migrations on it when switching to a new branch. When working longer on the same branch it should just be a matter of pulling in changes of other coworkers and if there's a migration file in the changes typing 'php migrate -m (file)' after the 'git pull' (or similar). 

[Michael Murphy]

Thanks for the reply and further clarification.

Would you recommend doing all field and template changes in code, and avoid using the UI? Is this something you would do from the start of building a PW project (completely avoiding the UI), or just after the initial build (using the UI)

I would be fine with that, and have always thought it would be nice to manage this in a text file. Do you have any more detailed example snippets of what this would look like?

I will be interested to hear how you use this with a fellow collaborator. One thing I am interested in is collaboration while developing, the other is maintenance / upgrade path for shared site profiles.

 

I am starting to see this module as a way to safely manage changes or updates for many common tasks that we sometimes have to do with PW. And I like the idea of having shared snippets, a bit like the PW recipes (maybe a new category for that site?). Another use case idea for multilingual sites would be a snippet that allows a site to easily change the default language by swapping all the field values.

[LostKobrakai]

The best possible way is probably using migrations throughout the whole development phase, but you can also add them in at a later stage as well. All you need is a common db (code) state to start from for (all) systems.

Personally I'd stay away from doing any template/field changes via the UI for as much as possible. This keeps you from "just changing this one thing real quick" and forgetting about it. But sometimes if you're experimenting because you're not 100% sure how to tackle a problem it might be faster to go that way. One just needs to be aware that everything needs to be a migration in the end. The second key is treating migrations like git branches/commits – Modify anything at will for as long as you haven't shared it to collaborators – Do not touch anything otherwise and rather add another migration to fix any errors/changes.

One benefit of having this level of detail in migrations (which might be daunting) is that you can really quickly apply/undo changes you've added to the system.

To get started with template/field changes you'd probably go like this (first two examples come with the module):

1. Create fields initially (example)
2. Create template initially (example) 
3. Make modifications with default migrations (example)
4. Repeat 1–3 as needed (Maybe delete something every now and then)

For snippets I've created the GitHub repo linked in above posts. I'll certainly add things there as well, but it's really meant to be a community effort to make this great. E.g. I'm currently not using multi-language anywhere, but others do so and might use interesting snippets.

I'm also very much interested in other opinions regarding the whole workflow. By now I'm using it just for myself, where it's working great (sadly I cannot share to much of it), but others might have different opinions and workflows.

[bernhard]

I'm still looking forward to trying migrations out for one project, but i have some other stuff to do and i think i will not be able to contribute anything before next year

but i already have a question  i understand the example https://github.com/LostKobrakai/Migrations/blob/master/migrations/2015-10-21_12-15.php as it is, but how would you create two fields? would you have to create a file for each field? and how would the workflow be if you made a mistake or forgot some settings on your field... for example set the limit of a file field to 1.

[LostKobrakai]

You'd create each field separately, so you can "rollback" each field separately, too. Migrations are not bound to "a specific development step" or alike. Just add as many as you need to bring the installation to the state you need.

Rollback's are also the way to go if you made some mistakes. Just rollback the migration, change your mistake and rerun it. If you've already shared the migration, or you've already saved data you want to keep, you could also add another migration, which does only correct your mistake, e.g. for your example just set the field limit and save this change.

 

[LostKobrakai]

I've updated the module with some small changes. Exceptions are no longer unintentionally catched when using the CLI tool. Also the default migration format does now include seconds in the timestamp. These are optional so migrations without seconds will still work.