Oh no, not another migration module!

[MarkE]

EDIT: Don't try this just yet, there seems to be a slight bug!

Version 0.0.4 on https://github.com/MetaTunes/ProcessDbMigrate

To upgrade, place the files in the module folder and refresh modules.

This version is partly a code tidy-up, but also adds 2 useful things to the module settings:

1. A (collapsed) help field which contains the help.md text
2. A feature which enables the current database to be named (e.g. Development, Test, Production, Client_1 or whatever). Migrations sourced from a named database will be treated as 'installable' in any database of a different name (or unnamed), but as 'exportable' in a database with the same name. This means, for example, that you can copy a production database as a new development database and rename it to be the same as your original development database, so that any migrations sourced from that development database will be shown as exportable, not installable, in the new database. You can also request that the current database name is notified in every admin page (in case you forget which environment you are in!). The feature is optional - if not used, any migrations will be treated as installable in every other database.

The module now does pretty much everything I originally wanted - it just needs a bit more use to flush out remaining bugs. It is also possible that further field types may be needed - there are some slight imperfections with pagetables and I do not have pro fields, so can't test those. And I am sure the code can be improved ? 

Edited April 17, 2021 by MarkE

[MarkE]

Hopefully the bugs in 0.0.4 have been fixed in v0.0.5

FWIW, the issue was to do with the operation of hooks when adding pages via a migration. 

The module was designed for migrating developments, not for mass updating of user pages. It was therefore assumed that any pages being migrated would be of a ‘site-settings’ nature, not user pages.

However, the module allows the migration of any pages and the host PW application may make use of page hooks. All page actions in the module allow hooks to run. To enable users to modify this behaviour, session variables are set for the duration of the following methods:
•    installPages() – new/changed pages – ‘dbMigrate_installPages’ is set to true
•    removeItems() – all item removals – ‘dbMigrate_removeItems’ is set to true
These can then be referenced in the application code as required. Note that in removals, all pages are trashed (so that hooks can operate) then deleted (so they are no longer in the trash).

So it is possible to use the module more generally (e.g. in ‘rescue’ mode) but test carefully first!

Please report any bugs (or indeed successful use). I'd also be grateful for feedback (particularly from module developers @adrian and @bernhard, who have previously commented, but also from any other interested persons) on whether this should be added to the modules library and, if so, whether anything needs to be done to it first.

[MarkE]

v0.0.6 allows page selectors that result in multiple levels.

The use of “sort=path” is permitted in selectors, even though this normally has no effect. If it is used, the pages will be sorted in parent-child order, using the ‘natural’ sort order (e.g. XYZ100 is greater than XYZ20). This means that parents should be installed before children. For ‘removed’ pages, the order is reversed so that children are deleted before parents.

[MarkE]

Latest version on github is v0.0.12. This fixes a number of bugs and also adds 2 new features:

• When a migration has been 'locked', so that it can no longer be changed/uninstalled etc., the 'preview' button is replaced with 'review'. The review shows all the changes made by the migration (on the assumption it was properly installed before locking); it is completely derived from the json files (which should be kept unchanged on the system) and therefore is completely independent of the current database state. BTW, the purpose of locking is to prevent inadvertant overwriting of other migrations if there is an overlapping scope (although warning messages are issued if this is a risk).
• Hopefully it is now (mostly) multi-language enabled. If anyone needs this and finds something is missing, let me know. Obviously you will need to add your own translations ? 

I've been using the module a fair bit and am pretty happy with it so far. It has been very useful when updating help pages (using @Macrura's excellent AdminHelp module) on the dev system - I can then release them as a consistent batch to the live system. I have also found that it is a good way of migrating language pages as it will migrate the language .json files automatically with the pages -  all that's needed is a migration item with selector "template=language".

[MarkE]

Version is now 0.0.13 - bug fixes and enhanced handling of image/file links in RTE fields.

BTW, @teppo, in PW weekly, characterises this module thus: "DbMigrate focuses more on migrating page content, while RockMigrations is more about the structure of the site". I'm not sure this is really correct: while I have made an effort to deal with page migrations, the module was not designed to be used for mass page migrations. The driver behind including page as well as template and field migrations was to facilitate migrations that cut across code, database structure and pages - typically involving settings-type pages which are maintained by the developer, not general users. If it assists in other page migrations as well then that is a beneficial side-effect, but be aware that I have only tested the page migrations with a limited set of field types (and no pro fields).

[MarkE]

v0.0.14 is now available

This is quite a significant enhancement in hookability so that now (for example) scripts can be run automatically after installing a migration. See the help file for full details. Brief outline is below.

Available hooks

Many of the ProcessDbMigrate methods are hookable:

• ___execute() – Display the Database Migrations setup page
• ___executeNewPage() – Create new migration
• ___executeGetMigrations() – refreshes all migration pages
• ___exportData($migrationPage) – creates the .json files for a migration
• ___removeFiles($migrationPage) – removes the .json files for a migration
• ___installMigration($migrationPage) – installs a migration in the target
• ___uninstallMigration ($migrationPage) - uninstalls a migration in the target
• ___lockMigration($migrationPage, $migrationFolder) – locks a migration (in the source)
• ___unlockMigration($migrationPage, $migrationFolder) – unlocks a migration (in the source)
• ___previewDiffs($migrationPage, $comparisonType) – previews migration changes where $comparisonType is one of: export, install, uninstall, review

Placement

You can place your hooks in site/ready.php. In this case you will need to check the name of the migration page before running – e.g.

wire()->addHookAfter('ProcessDbMigrate::installMigration', function ($event) {
    $migrationPage = $event->arguments(0);
    if ($migrationPage->name == 'my-migration') {
        ///code to run
    }
});

and then you use $migrationPage to further reference the migration page. This approach keeps all your hooks together and you have to remember to sync the site/ready.php as well as your migration files.

Alternatively (the recommended approach), you can place a file called ready.php in the site/template/DbMigrate/migrations/my-migration/ directory, in which case your script would be

$this->addHookAfter('ProcessDbMigrate::installMigration', function ($event) {
// code to run
});

and you can use $this to reference the migration page. This approach keeps all your migration-related data & code together and you only have to sync the migration folder. It also means that your migration-specific ready.php code will be displayed at the bottom of the migration page. Also, if you ‘remove migration files’ it will remove your custom code as well (usually you will only be doing this as preparation to delete the migration, so that is what you want). With the first approach, your hook will remain in site/ready.php.

Usage

Of the available hooks, installMigration and uninstallMigration are likely to be the most useful. For example, a hook after ProcessDbMigrate::installMigration could be used to carry out database-wide changes following the amendment of the structure. Say you have changed an address field to split out the post code into a separate field. The migration definition will specify the new field and the changed template. The hook (after ProcessDbMigrate::installMigration) will then include the code to extract postcodes and place them in the new field. You could place code to undo this as a hook before ProcessDbMigrate::uninstallMigration, so that executing the uninstall exactly reverses the install.

---------------

UPDATING - NOTE

If you are updating the module (as opposed to installing for the first time), after a 'modules refresh' you will need to go to the bootstrap migration page and install it in order to pick up the new field which displays the migration-specific ready.php file.

 

[dotnetic]

Thanks for your continued development on this. I still need to have a look at it.

[MarkE]

13 minutes ago, dotnetic said:

Thanks for your continued development on this

It’s now pretty much “finished” - in the sense that I don’t have much more planned in the way of features (unless others make suggestions), although I’m sure there will still be bug fixes?

There are just a couple of possible additions: allowing selectors for fields and templates as well as pages, and providing some sort of database comparison feature. 

[antpre]

Hi,

Tks for the module. Wanted to install it to give it a try.
I got this error message when the installation started.

I have installed the Fieldtype Runtime Only module before.

do you have any suggestion ?

I am running it on ProcessWire 3.0.164 and php 7.4

Thanks a lot

 

[antpre]

Hi,

Sorry for the previous message. It seems to be my environment that is not working fine. I could set up the module in two other environments.

Got a warning message (see screen shot below). Same message in both case. But module was installed but the dbmigration page were not installed. So not realy usable. Is there something I am missing in the installation process ?

Regards

[MarkE]

On 6/7/2021 at 2:03 PM, antpre said:

Got a warning message (see screen shot below). Same message in both case. But module was installed but the dbmigration page were not installed. So not realy usable. Is there something I am missing in the installation process ?

Hi @antpre. Sorry - I've been away for a few days - will look into this.

[MarkE]

Hi @antpre. There seems to be a couple of issues here:

1. The first exception is a bug, I think. Please download the current version of DbMigrationPage.class.php which should fix it.
2. The second problem is caused because your module folder is called ProcessDbMigrate-master, not ProcessDbMigrate (this is just because of GitHub download). So: 
   1. Get the new version of DbMigrationPage.class.php as per above
   2. Uninstall the module
   3. Rename the module folder to ProcessDbMigrate
   4. Refresh modules
   5. Install the module

I hope that works. Let me know if not and if you have any other queries.

[antpre]

Hi MarkE,

Thanks a lot for your response. with the latest DbMigrationPage.class.php eveything went fine.

I was impatient to try this module that adds a  set of features that I was realy missing in PW.

Hope the module will go on the PW module page at some point.

Thanks again. Bye

[MarkE]

For @antpre and anyone else trying this module - there is a minor bug in the presentation - please update RuntimeOnly/dbMigrateReady.js and copy it to your templates/RuntimeOnly folder.

[antpre]

Hi, I ve been testing the module a bit today.

Could export/import : fields (date, text,page ref), templates, pages. It worked well. Great

However I have had a problem with repeater fields. I get this error message when I add a repeater field migration item at the time when i click the "export data" button. I am a bit confused cause no template are related to repeater field. I m not understanding if i am doing something wrong.

I first tried a migration with all the fields included in the repeater plus the repeater field itself. Then I tried a migration with only a few repeater fields. Same result.

In the screen shoot each item is a repeater field (title_workflow...).
Any idea.
Thanks a lot

[MarkE]

Hi @antpre. Could you post a screenshot of the migration page showing the migration items (open)?

[antpre]

Hi @MarkE,

here is the screen shot.

regards

[MarkE]

Hi @antpre. It is not obvious, but every repeater field has a system template which is the field name preceded by repeater_   . If you go to setup/templates and enable system templates in the filter you will see these. You therefore need to include the details of this template in your migration, before the field itself - as indicated by the message. Also, if your repeater uses any other new or changed fields, you will need to specify these before the template. 

[antpre]

Thanks a lot MarkE,

I had no idea that there was a system template created for every repeater field (it make sense when you think about it).

It works fine.
Thanks a lot for your support and the module.

[antpre]

Sorry to come back on the issue again.
It works fine when there is only one repeater field in the db migration. As soon as I add more than one the warning message comes again (I have tested these repeater independently in one db migration and then it works).

[MarkE]

I’ll look into that @antpre, but it is possible that the warning message is being shown in error. It may be that the migration still works ok. Let me know either way. BTW I don’t see any migration items for the fields in the repeater templates - that’s ok if they already exist in the target, but please check. 

[cjx2240]

Hi, I really wanted to try this out but I hit a roadblock straight away... the page didn't appear in the CMS after installation. I made sure the module folder was named correctly (saw the similar issue above) and I have the latest version of the code

I also couldn't uninstall the module, it hit a php error.

While trying to figure it out I've been uninstalling/reinstalling, it definitely hits various issues with the fact it can't remove/update system templates.

It looks very interesting though, hope I can figure it out and have a go!

Edit: ahhh well, I see the problem. Your bootstrap for creating the admin page assumes that the admin URL will be "/processwire/" - but mine is not! May be better to grab the admin page name there

Edit 2: well I have had a look at it and I really like the idea of the way it works! Bit rough around the edges for sure (one bug I seem to have noticed is that I get errors about a sixth repeater row repeatedly after I have deleted it, same thing happens with snippets. They're gone from the original db, but visible and empty when viewing the migration on the target db more on that later*) but the idea is much more in line with how I would want to do migrations.

For me though I hit two snags in my first attempt to use it. One is that it didn't include a hidden page in the export data, nor did it allow me to add something like "include=all" to the selector, so I could only add it by un-hiding my page to create the migration. Another is that it didn't include data from a ProFields: Table field, but I can populate that with a hook which is a great addition!

Edit 3: * I actually hit a pretty big bug with this later on. I tried (mistakenly) to remove the errant rows on the target db and save it. I got the message saying that I can't actually save changes on the target db (fair enough) but then the status became "superseded" and I could no longer perform the migration.

[MarkE]

Thanks for the comments @cjx2240. Sorry it has taken so long to reply. I have done quite a bit of work removing various bugs (there were quite a few!). I have also extended the functionality considerably to include a database comparison feature (allows draft migrations to be created based on database differences). There is also the capability to run custom code alongside the migration via hooks. The help file is now (roughly) indexed and available at https://metatunes.github.io/DbMigrate/help.html. The new version (0.1.0) is at https://github.com/MetaTunes/ProcessDbMigrate. You should be able to replace the files in your ProcessDbMigrate directory with the new ones and 'refresh modules' to upgrade.

Please let me know if any of the snags you hit have persisted after the update (or any new ones encountered) and I will try and resolve them more quickly next time! I'm hoping that this version is a passable 'alpha' as opposed to the previous 'proof of concept' and will look to submit it to the modules directory if it seems sufficiently robust.

[cjx2240]

Hey glad to see the update! Things are also very busy on my end but I definitely want to give this a good look soon. Sounds like all good things.

[MarkE]

I found a few more small bugs and have also greatly enhanced the help file, so that it is now available as a pop-out when working in the module. The new version (0.1.3) is at https://github.com/MetaTunes/ProcessDbMigrate. You should be able to replace the files in your ProcessDbMigrate directory with the new ones and 'refresh modules' to upgrade.

@cjx2240, @antpreand @dotnetic (and anyone else): I'd appreciate your feedback as I am considering releasing this (as alpha) to the modules directory.