ProcessDbMigrate

[MarkE]

Season's Greetings ProcessWirers! I hope you enjoy the gift of this module, but use with care...

TLDR: This module captures changes made in the development environment so that they can be easily migrated to the live environment without needing to specify the changes or write any code. The demo below gives a brief overview.

Want to read? Read on.

One of the (few) problems with ProcessWire, in my opinion, is the lack of any native way of handling migrations. Given that PW is such a powerful tool capable of sophisticated and complex web-based applications, this is less than ideal. There is a solution, however, in RockMigrations which accomplishes a lot in a controllable way, provided you are happy to specify your database set-up in code rather than via the UI (albeit that the latest versions allow you to grab much of the required code from the UI). If that suits your need, great. Around the same time as the first versions of RockMigrations, I started developing my own UI-based migrations module, which I have been using with reasonable success for some time. I halted development of the module for a while as RockMigrations developed and I considered switching to that route.

However, I decided that my module suited me better and that a real improvement could be made if it was effectively automated so that I no longer needed to specify a migration. So that is exactly what it does: after configuring the module, you add a new migration page with ‘log changes’ enabled (which includes determining what types of objects are relevant for the migration) and work on your development system. Once you have made the desired changes (and tested them!) in the development environment, you go back to the migration page where it has magically captured the objects which have changed and listed them in dependency order. You then ‘export’ the changes, which creates json files to be uploaded to the live environment (via Git or FTP etc.), where they are then ‘installed’ to re-create the changes in the live system.

The demo below illustrates this briefly. This first demo shows the creation of a migration. The installation demo will be in the next post, because of size constraints. See post 4 for HD video.

 

There is a very extensive manual which covers all the features of the module, not just this ‘automatic’ method.

Available on github at https://github.com/MetaTunes/ProcessDbMigrate and in the modules library here.

PLEASE NOTE that this is still in 'alpha'. Do not use in production without fully testing and backing up at every stage. It is quite complex so, although I have tried hard to eliminate bugs, there will inevitably be some left!

Edited December 23, 2023 by MarkE
Now in modules library

[MarkE]

Demo part 2, showing the installation of the migration was going to be here, until I realised that the file constraint is for the thread as a whole!

You can see it here: ProcessDbMigrate video - target - 1703029659248.mp4

See post 4 for HD video.

[szabesz]

Nice work and thank for sharing @MarkE! Could you please somehow share higher-resolution videos than those above? (Maybe using YouTube?) The first one is 37MB, and the second one is only 27MB, making the text hard to read. Sure, I understand what is going on, but still...

[MarkE]

1 hour ago, szabesz said:

Could you please somehow share higher-resolution videos than those above?

Sure. Part 1 is here: https://1drv.ms/v/s!AmS0-Sk4lDz9h5tDqJwj50qq6-KEhg

and part 2 here: https://1drv.ms/v/s!AmS0-Sk4lDz9h5tS2DJsT94F92fJmw

If you want to play with it over the holidays ?, the best thing to do is to create a clean PW installation - ddev is great for doing this quickly - and I would recommend add ProcessDatabaseBackups and TracyDebugger. Start by following similar steps to the video.

Note that, in the video, I have 'simulated' the migration installation by taking a backup before adding new fields etc. Take another backup after creating the migration, then restore the original one, changing the database name in the module config.

And don't forget to RTM: https://metatunes.github.io/DbMigrate/help.html

Edited December 20, 2023 by MarkE
further info

[szabesz]

Thanks @MarkE!

35 minutes ago, MarkE said:

If you want to play with it over the holidays

I will! :)

[gebeer]

Thank you for this module. Great addition. Now people who prefer to work more in the GUI can manage migrations with your module. The automatic recording feature looks interesting.

[MarkE]

Now in the modules library at https://processwire.com/modules/process-db-migrate/. 

[valan]

After 10 years of PW development it looks like a break-through!? @MarkE thank you! I'll consider using PW again in my next prj.

[MarkE]

Version 2.0.16 now released. This comprises mosly bug fixes after extensive testing on some really complex migrations involving many templates fields and pages. In particular, the module will automatically carry out multiple migration installations to deal with the problem of circular dependencies between changed objects.

Not sure how many people are using this module, but any feedback is welcome.

[Flashmaster82]

I can´t install this module

Fatal Error: Uncaught Error: Call to a member function add() on null in site/modules/SkipInputfieldLabel/SkipInputfieldLabel.module.php:41

[MarkE]

5 hours ago, Flashmaster82 said:

Fatal Error: Uncaught Error: Call to a member function add() on null in site/modules/SkipInputfieldLabel/SkipInputfieldLabel.module.php:41

I’m not familiar with the SkipInputfieldLabel module, which is the source of the error. It looks like there is some sort of interaction with ProcessDbMigrate, but I think that there should be some error checking in SkipInputfieldLabel. Perhaps @teppo can help? You could try temporarily uninstalling SkipInputfieldLabel but I’m not sure if that would cause a problem with that module. 

[Flashmaster82]

On install PW 3
( ! ) Error: Call to undefined function ProcessWire\bd() in E:\Wamp\www\Test1\site\modules\ProcessDbMigrate\ProcessDbMigrate.module.php on line 693

[MarkE]

4 hours ago, Flashmaster82 said:

Call to undefined function ProcessWire\bd()

Looks like I forgot to take out a debugging statement and that gives an error if TracyDebugger is not installed. I am planning a new release shortly which should fix that and a few other bugs. Meanwhile @Flashmaster82, if you took a database backup before the installation, you may wish to revert to it so that next time you get a clean install. It may be a couple of days before the next version, so bear with me. BTW, I really do recommend TracyDebugger (but it shouldn't be a requirement!).

[MarkE]

v2.0.21 released, which I hope fixes your problem @Flashmaster82. Let me know how it goes. Many other bug fixes are included and particular thanks to @Jonathan Lahijani for many of these.

[bernhard]

@Jonathan Lahijani as I know you are (or have been) a RockMigrations user I'm wondering whether you switched to ProcessDbMigrate or you are using both modules in parallel? Would be interesting to hear the reasons for your decision and how everything went or what you are using which module for.

[Jonathan Lahijani]

@bernhard RockMigrations and ProcessDbMigrate take fundamentally different approaches.  RM takes the traditional web application framework approach while ProcessDbMigrate is more of a change recorder, similar to Craft CMS Project Config.  There's trade-offs to each approach, but I will say it's nice to have changes to be recorded automatically, even if it's not a 100% full-proof approach (although I'm looking into that).  It's basically automating what I typically do when I need to migrate a changes "by hand" on the sites I work on, which is an approach I like.

I'm experimenting with the change recorder approach in a module of my own for now, but development of it is on and off for the moment.  Understanding ProcessDbMigrate helped with understanding the approach, and the recording part of it is straight-forward.  The "playing" of the recording gets tricky for all the reasons we all know about.

[Flashmaster82]

Error installing module - ProcessDbMigrate
ModulesInstaller: Unable to install module (ProcessDbMigrate): You must assign a template to the page before setting field values (dbM)

[MarkE]

56 minutes ago, Flashmaster82 said:

Unable to install module (ProcessDbMigrate): You must assign a template to the page before setting field values (dbM)

Hmm. I thought I had fixed that.... Bear with me 😁

[MarkE]

Hi @Flashmaster82, v2.0.22 released which I hope finally addresses your issue. Please let me know how it goes (and remember, it's still alpha, so take backups at all key points, such as before installing new versions and before installing migrations).

[Flashmaster82]

@MarkE  Thank you so much.

Edited September 24 by Flashmaster82

[Flashmaster82]

When installing

• Bootstrap failed
• Missing parent or template for page "/Test1/processwire/dbmigrations/". Page not created/saved in this installation attempt (may be achieved later).
• Missing parent or template for page "/Test1/processwire/dbcomparisons/". Page not created/saved in this installation attempt (may be achieved later).

After uninstall

Fields: Fieldtype module 'FieldtypeDbMigrateRuntime' for field 'dbMigrateRuntimeReady' is missing (Can´t get rid of this error message in the top).

Edited September 24 by Flashmaster82

[MarkE]

1 hour ago, Flashmaster82 said:

Bootstrap failed

Hmm. Not sure why that is. I tested it on a clean site and it all worked smoothly. Do you have any more details?  Did you keep a backup before the installation - if so, were there any pages/fields/templates from previous failed ProcessDbMigrate installation attempts? (Check system templates and fields and pages in /admin/ and /admin/setup/)

 

[Flashmaster82]

If installed correctly, should there be a link of sort in the left tree menu under setup for settings or similar? I tried to install the module again and there where a couple of errors but its in the module directory. But i dont see any link to the module besides in the module directory.

Edited September 24 by Flashmaster82

[MarkE]

2 hours ago, Flashmaster82 said:

If installed correctly, should there be a link of sort in the left tree menu under setup

There should be a link under setup. 

[Flashmaster82]

48 minutes ago, MarkE said:

There should be a link under setup. 

I guess it did not installed correctly then.. Also when uninstall the module templates are still in the system and can´t be removed because its system templates.

Edited September 24 by Flashmaster82