RockMigrations1 - Easy migrations from dev/staging to live server

[MrSnoozles]

@bernhardAny idea already what will be different in the new module?

Also it feels like it's not encouraged to use the old version anymore, because the Readme has been stripped from the repository. Is that feeling correct?

[bernhard]

The main difference is how migrations are triggered. On the old module I've used the "fireOnRefresh" approach which fired migrations on every modules refresh. On the new version you can add files to a watchlist and migrations will automatically fire if a file changed. That makes a huge difference when working with the module as you'll only need a page reload and the new data will be there ? 

Also I've improved logging so that it's easier to see what happens and the module handles same cases in the same way (the old one sometimes threw exceptions, sometimes did early exists).

Other than that it's quite the same as the old one as I'm really happy with how everything worked there. I've refactored all methods while transferring them from the old version to the new one.

[MrSnoozles]

I've played with RockMigrations over the last week to get a better feeling of what it does.
I didn't like the approach at first, because imo migrations should be run once to bring the system to the required state. Having them run on module refresh seems a bit too dangerous for production sites.

I agree though, that for the sake of getting things done, your approach is very convenient. What you have planned for the next version could combine the best of both worlds:
- a directory is added to the watchlist
- migrations in that directory are executed in an alphabetical order
- a hash-value for the file contents is stored somewhere when the migration is run
- as long as the file contents does not change, the migration is not executed anymore

That way the quick protoyping, as well a "proper" 1-file-per-update migrations could be achieved. Is this what you have in mind too?

[bernhard]

45 minutes ago, MrSnoozles said:

- a hash-value for the file contents is stored somewhere when the migration is run
- as long as the file contents does not change, the migration is not executed anymore

No. All migrations will run again, but if you don't change anything they will likely not do anything. The good thing about this is that if something changed for whatever reason, the migration will change things back to the state that is defined in code.

So for example if someone changed a field setting by hand, the migration would revert this change to what is stored in the migration.

To make it more obvious that it might not be the best idea to change field or template settings by hand the new version stores a trace of the migration for every field and template and shows a warning on the field/template editor:

[MrSnoozles]

Ah, good point. Thanks for the clarification. Looking forward to seeing what you're coming up with.

[elabx]

On 3/21/2022 at 2:49 AM, MrSnoozles said:

- a directory is added to the watchlist
- migrations in that directory are executed in an alphabetical order
- a hash-value for the file contents is stored somewhere when the migration is run
- as long as the file contents does not change, the migration is not executed anymore

@MrSnoozles You could take a look at using Migrations + RockMigrations (which work great hand by hand!) which satisfies exactly this. So you only substitute the "migration files" system, but use the invaluable abstractions from RockMigration.

@bernhard interesting idea the tracking within the fields!

[bernhard]

Hey @elabx could you please describe how you are using RockMigrations and why you are using it in combination with Migrations? That would be very interesting for me and maybe also others ? 

[elabx]

45 minutes ago, bernhard said:

Hey @elabx could you please describe how you are using RockMigrations and why you are using it in combination with Migrations? That would be very interesting for me and maybe also others ? 

So it goes like this, you create a migration file under the Setup > Migrations. This is a "Default" type of migration. Which is the one I only use, since with RockMigrations the whole API work is abstracted so nicely that working with the "data schema" (meaning ALL the config in processwire) is really more simple, so no need for the specific migrations types.

<?php

class Migration_2022_06_02_11_42_58 extends Migration {

	public static $description = "Do some awesome migrations stuff";

	public function update() {
	  // Put your migration code here
      $rm = wire('modules')->get('RockMigrations');
      $rm->migrate([
		'fields' => 
			[
				'button_label' => ['label'=> 'Button Label', 'type' => 'text']
			] 
	  ]);
	}

	public function downgrade() {
		// Put your rollback code here
	}

}

Then this runs either through the UI under Setup > Migrations or the CLI included in the Migrations module.

We could say one "caveat" is that to use the CLI you need to install the Migration module through composer as the CLI tool has dependencies assuming them. 

So every time I wan to to push a migration to the live site, the pipelines I use for deployment (Bitbucket Pipelines) rsync's the new files, then triggers on the server:

php /path-to-website/site/modules/Migrations/bin/migrate run

Or for anyone reading this not familiarized with CI/CD pipelines, just log into the server and run the command. So "migrate run" runs all "pending" migrations. Previously run migrations are already tracked as migrated, so they won't run again. cc @MrSnoozles.

I DO NOT, have an automated rollback solution yet, in terms of the CI/CD pipeline. So if I break something, I just move forward with another migration.

Let me know if it sounds I'm skipping something, wrote this a bit quickly!

[bernhard]

Thx for the quick and detailed answer ? Have you seen the new version? https://github.com/baumrock/rockmigrations#running-migrations

What you describe is possible with a single file called /site/migrate.php (similar to /site/ready.php for hooks): https://github.com/baumrock/rockmigrations#quickstart

No need for another module, no need for creating separate php files with php class notation for simple migration changes, no need for a CLI. Unless I'm missing something ? 

[elabx]

On 6/3/2022 at 12:39 PM, bernhard said:

Thx for the quick and detailed answer ? Have you seen the new version? https://github.com/baumrock/rockmigrations#running-migrations

I  had not! ? Thanks!

On 6/3/2022 at 12:39 PM, bernhard said:

What you describe is possible with a single file called /site/migrate.php (similar to /site/ready.php for hooks): https://github.com/baumrock/rockmigrations#quickstart

No need for another module, no need for creating separate php files with php class notation for simple migration changes, no need for a CLI. Unless I'm missing something ? 

Let me read the module's readme and I might come back with questions ?

[bernhard]

3 hours ago, elabx said:

Let me read the module's readme and I might come back with questions ?

Yeah that's the part that is missing and why I've not yet officially released it. I want to prepare better docs and an intro video to make it as easy to use as it can get. Any help in that direction is welcome, so if you try it please let me know all your questions, concerns etc ? 

[BlindPenguin]

@bernhardI'm also trying it right now and so far i love it. I do have a few questions, though.

What is the recommended function to migrate? I see mentions of "createField" and "migrate" at other places, so i'm a bit confused. And does "migrate" update field settings, when i change something like the "tags" of a field?

The other thing: How does it work with translations? Can i use the default translation modules that come with ProcessWire?

EDIT: Since I'm just playing around with options: How do i translate them? "options1234" doesn't seem to work.

Edited June 13, 2022 by BlindPenguin
Random thought

[bernhard]

On 6/13/2022 at 6:51 PM, BlindPenguin said:

@bernhardI'm also trying it right now and so far i love it. I do have a few questions, though.

Hi @BlindPenguin glad to hear that ? 

On 6/13/2022 at 6:51 PM, BlindPenguin said:

What is the recommended function to migrate? I see mentions of "createField" and "migrate" at other places, so i'm a bit confused. And does "migrate" update field settings, when i change something like the "tags" of a field?

It does not really matter as the migrate() method uses createField() and setFieldData() etc behind the scenes. What might make a difference is that migrate() makes sure that all fields and all templates exist before it sets their data. That means you can reference fields and templates from within that migrate call and they can even reference each other without creating an infinite loop!

On 6/13/2022 at 6:51 PM, BlindPenguin said:

The other thing: How does it work with translations?

It depends.

On 6/13/2022 at 6:51 PM, BlindPenguin said:

Can i use the default translation modules that come with ProcessWire?

Of course you can. RockMigrations does not care.

On 6/13/2022 at 6:51 PM, BlindPenguin said:

EDIT: Since I'm just playing around with options: How do i translate them? "options1234" doesn't seem to work.

At the moment RockMigrations can't do that. What's interesting is that even the core field export does not export translations of an options field. So I'm not sure if that is possible at all at the moment.

For a real world use (aka quickfix) you could just create the field via RockMigrations, push the migration to the live system (which will create the field), then translate your options and then pull the  DB changes to your dev environment (manually).

If you need a 100% automated solution you could use pages and a page field to select the option (that's the way we did it before the options field even existed).

Does that help?

[BlindPenguin]

@bernhard

Quote

What's interesting is that even the core field export does not export translations of an options field.

That's weird. I noticed in the database that it seems to create the columns "title1234" and "value1234" automatically.

Quote

For a real world use (aka quickfix) you could just create the field via RockMigrations, push the migration to the live system (which will create the field), then translate your options and then pull the  DB changes to your dev environment (manually).

But doesn't RockMigrations overwrite that immediately after refreshing modules?

I guess it's possible to check if the field already exists, then add the translations with some database magic. I've seen some promising functions in the ProcessWire folders. Maybe i can work something out.

And yes, all that information helps greatly. Thanks! ?

[bernhard]

9 hours ago, BlindPenguin said:

But doesn't RockMigrations overwrite that immediately after refreshing modules?

No, if you just do a createField it will just update the field's label and it will not touch its values. You could do something like this:

<?php
// site/migrate.php
/** @var RockMigrations $rm */
$rm = $this->wire->modules->get('RockMigrations');
$rm->createField("yoursettingsfield", "options", [
  'label' => 'Your field label',
  'tags' => 'MyWebsite',
  // values are populated manually on the live system
]);

Push that to the live system, populate values, pull DB to dev.

RockMigrations works in a way that it only sets properties that are listed in the migration config. That means if you REMOVE properties, it will most likely not remove those values from the DB. That's why the template migrations have a special property "fields-" additional to "fields":

<?php
...
$rm->setTemplateData('yourtpl', [
  'fields' => ['foo', 'bar', 'baz'],
]);

// this will NOT remove field baz from template yourtpl
$rm->setTemplateData('yourtpl', [
  'fields' => ['foo', 'bar'],
]);

// this WILL remove field baz from template yourtpl
$rm->setTemplateData('yourtpl', [
  'fields-' => ['foo', 'bar'],
]);

// but I prefer doing such things like this as it's more verbose and clearer
$rm->setTemplateData('yourtpl', [
  'fields' => ['foo', 'bar'],
]);
$rm->removeFieldFromTemplate('baz', 'yourtpl');

Of course you can also do conditional migrations like this one where I had to migrate from this structure

/rockmails
/rockmails/mail1
/rockmails/mail2
...

to this one:

/rockmails
/rockmails/templates
/rockmails/templates/tpl1
/rockmails/templates/tpl2
...
/rockmails/mails
/rockmails/mails/mail1
/rockmails/mails/mail2
...

The migration creates the new root data page (/rockmails - tpl=rockmails_root), then it creates the mails and mail template and fields, then, if the old mailspage exists (/rockails - tpl=rockmails), it moves the old mailspage from PW root to the new datapage (/rockmails/mails) and it renames it from rockmails to mails (as it lives in /rockmails/mails now and /rockmails/rockmails would be weird) and when done we call the root->migrate() again to revert the setTemplateData from line 200

---

If you find something out regarding to the options field please let me know. I'm happy to merge PRs ? 

[BlindPenguin]

@bernhard Ok, so my ProcessWire knowledge isn't exactly up to the task to do a PR for your module. But i was able to use PDO statements to work around it.

Quote

$rm->createField('risk_level', 'options', [
    'label' => 'Risk Level',
    'label1024' => 'Risiko Level',
    'type' => 'options',
    'options' => [
        1 => 'Very Low',
        2 => 'Low',
        3 => 'Middle',
        4 => 'High',
        5 => 'Very High',
    ],
]);


if ($fields->get('risk_level')) {
    $riskLevel = $fields->get('risk_level');
    $options = [
        1 => 'Sehr niedrig',
        2 => 'Niedrig',
        3 => 'Mittel',
        4 => 'Hoch',
        5 => 'Sehr hoch',
    ];

    foreach($options as $key => $option) {
        $query = $database->prepare("UPDATE fieldtype_options SET title1024='$option' WHERE option_id='$key' AND fields_id='$riskLevel->id'");
        $database->execute($query);
    }
}

It might not be beautiful, but it works.

I've seen several functions in the SelectableOptionsManager, like the "setOptionsStringLanguages" function. My guess is, that this one is used in the backend for saving the translations. I couldn't work out yet how to use it properly. But i think that would be the most promising function.

[bernhard]

Hey @BlindPenguin I've added support for translatable options fields in RockMigrations. It was quite tricky but should work now. I'd be happy if you could do some testing and tell me if everything works as expected.

<?php
$rm->createField("tmp_opt", "options", [
  'label' => 'testing options field migration',
  'options' => [
    1 => 'one|Label for Option one',
    2 => 'two|Label for Option two',
    3 => 'three|Label for Option three',
  ],
  'optionLabels' => [
    'german' => [
      'Beschriftung für Option eins',
      'Beschriftung für Option zwei',
      'Beschriftung für Option drei',
    ],
  ],
]);

 

[BlindPenguin]

Sorry, just saw your message today. I tried it with a new field and also tried uninstalling and reinstalling the module beforehand. It does fill the German field, but it also duplicates the values in the original one. I have two languages configured: "default" and "de".

$rm->createField('test_field', 'options', [
    'label' => 'Test Field',
    'label1024' => 'Test Feld',
    'type' => 'options',
    'options' => [
        1 => 'VERYLOW|Very Low',
        2 => 'LOW|Low',
        3 => 'MIDDLE|Middle',
        4 => 'HIGH|High',
        5 => 'VERYHIGH|Very High',
    ],
    'optionLabels' => [
        'de' => [
            'Sehr niedrig',
            'Niedrig',
            'Mittel',
            'Hoch',
            'Sehr hoch',
        ],
    ],
]);

The result seems a bit different, though:

 

When refreshing the modules, i get the following error:

"title may not contain the character '|'".

 

[bernhard]

can you try to create the field manually and then copy and paste the field code from the first tab?

[BlindPenguin]

For some reason i stopped getting notifications...

Manually creating fields shows me this code:

'manual_select_field' => [
  'label' => 'Manual Select Field',
  'flags' => 0,
  'type' => 'FieldtypeOptions',
  'label1024' => 'Manuelles Select Feld',
  'inputfieldClass' => 'InputfieldSelect',
  'collapsed' => 0,
  'tags' => '',
  'initValue' => '',
  'showIf' => '',
  'themeInputSize' => '',
  'themeInputWidth' => '',
  'themeOffset' => '',
  'themeBorder' => '',
  'themeColor' => '',
  'columnWidth' => 100,
  'required' => '',
  'requiredIf' => '',
  'defaultValue' => '',
  'options' => [
    1 => 'VERYLOW|Very low',
    2 => 'LOW|Low',
  ],
  'optionLabels' => [
    'de' => [
      1 => 'VERYLOW|Sehr niedrig',
      2 => 'LOW|Niedrig',
    ],
  ],
],

When i use that in my migration code for another field (with the migrate command), the duplication effect also happens in the German field. When i only give the German labels, then i have the same effect as before. What's also interesting: When i remove the values "VERYLOW" and "LOW" from both fields, including the "|", then the duplication doesn't happen.

That strange error message while refreshing is still there. I think i'll try a clean installation, just in case i messed something up. I'll look into it tomorrow.

[BlindPenguin]

Ok, i now have an entirely new setup. It has the same problem. And i just thought about turning on debug. That might help. ? I avoid that option usually, because it showers me in a myriad of deprecation errors.

First Error:

SelectableOption: title may not contain the character '|'

-------------------

Second Error:

ProcessWire: ProcessModule: Too many keys: PDO::quote(): Passing null to parameter

DEBUG MODE BACKTRACE ($config->debug == true):
#0 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(1124): Spyc->checkKeysInValue()
#1 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(609): Spyc->returnKeyValuePair()
#2 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(545): Spyc->_parseLine()
#3 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(494): Spyc->loadWithSource()
#4 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(165): Spyc->_load()
#5 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/YAML.php(11): Spyc::YAMLLoad()
#6 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/RockMigrations.module.php(3025): RockMigrations\YAML->load()
#7 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/RockMigrations.module.php(1802): ProcessWire\RockMigrations->yaml()
#8 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/RockMigrations.module.php(1979): ProcessWire\RockMigrations->migrateWatchfiles()
#9 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/RockMigrations.module.php(2739): ProcessWire\RockMigrations->run()
#10 /home/ferdl/PhpstormProjects/process/test-project/wire/core/WireHooks.php(1059): ProcessWire\RockMigrations->triggerMigrations()
#11 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(485): ProcessWire\WireHooks->runHooks()
#12 /home/ferdl/PhpstormProjects/process/test-project/wire/modules/Process/ProcessModule/ProcessModule.module(368): ProcessWire\Wire->__call()
#13 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(414): ProcessWire\ProcessModule->___execute()
#14 /home/ferdl/PhpstormProjects/process/test-project/wire/core/WireHooks.php(951): ProcessWire\Wire->_callMethod()
#15 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(485): ProcessWire\WireHooks->runHooks()
#16 /home/ferdl/PhpstormProjects/process/test-project/wire/core/ProcessController.php(337): ProcessWire\Wire->__call()
#17 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(414): ProcessWire\ProcessController->___execute()
#18 /home/ferdl/PhpstormProjects/process/test-project/wire/core/WireHooks.php(951): ProcessWire\Wire->_callMethod()
#19 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(485): ProcessWire\WireHooks->runHooks()
#20 /home/ferdl/PhpstormProjects/process/test-project/wire/core/admin.php(160): ProcessWire\Wire->__call()
#21 /home/ferdl/PhpstormProjects/process/test-project/site/templates/admin.php(18): require('...')
#22 /home/ferdl/PhpstormProjects/process/test-project/wire/core/TemplateFile.php(327): require('...')
#23 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(414): ProcessWire\TemplateFile->___render()
#24 /home/ferdl/PhpstormProjects/process/test-project/wire/core/WireHooks.php(951): ProcessWire\Wire->_callMethod()
#25 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(485): ProcessWire\WireHooks->runHooks()
#26 /home/ferdl/PhpstormProjects/process/test-project/wire/modules/PageRender.module(575): ProcessWire\Wire->__call()
#27 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(417): ProcessWire\PageRender->___renderPage()
#28 /home/ferdl/PhpstormProjects/process/test-project/wire/core/WireHooks.php(951): ProcessWire\Wire->_callMethod()
#29 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(485): ProcessWire\WireHooks->runHooks()
#30 /home/ferdl/PhpstormProjects/process/test-project/wire/core/WireHooks.php(1059): ProcessWire\Wire->__call()
#31 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(485): ProcessWire\WireHooks->runHooks()
#32 /home/ferdl/PhpstormProjects/process/test-project/wire/modules/Process/ProcessPageView.module(184): ProcessWire\Wire->__call()
#33 /home/ferdl/PhpstormProjects/process/test-project/wire/modules/Process/ProcessPageView.module(114): ProcessWire\ProcessPageView->renderPage()
#34 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(417): ProcessWire\ProcessPageView->___execute()
#35 /home/ferdl/PhpstormProjects/process/test-project/wire/core/WireHooks.php(951): ProcessWire\Wire->_callMethod()
#36 /home/ferdl/PhpstormProjects/process/test-project/wire/core/Wire.php(485): ProcessWire\WireHooks->runHooks()
#37 /home/ferdl/PhpstormProjects/process/test-project/index.php(55): ProcessWire\Wire->__call()
#38 {main}

I may have to add that i'm on Manjaro Linux, which is using PHP 8.1.7 by default. Switching to PHP 7.4 appears to get rid of the PDO error. But the SelectableOptions error and the strange duplication behavior remains.

[bernhard]

Hey @BlindPenguin I think I got it working now. Can you try the latest version? Note that I changed the syntax!

<?php namespace ProcessWire;
/** @var RockMigrations $rm */
$rm = $this->wire->modules->get('RockMigrations');
$rm->createField('test_field7', 'options', [
  'label' => 'Test Field',
  'label1020' => 'Test Feld',
  'type' => 'options',
  'optionsLang' => [
    'default' => [
      1 => 'VERYLOW|Very Low',
      2 => 'LOW|Low',
      3 => 'MIDDLE|Middle',
      4 => 'HIGH|High',
      5 => 'VERYHIGH|Very High',
    ],
    'de' => [
      1 => 'VERYLOW|Sehr niedrig',
      2 => 'LOW|Niedrig',
      3 => 'MIDDLE|Mittel',
      4 => 'HIGH|Hoch',
      5 => 'VERYHIGH|Sehr hoch',
    ],
  ],
]);

[BlindPenguin]

@bernhardNow it's working perfectly. The "|" error is now gone. In PHP 7.4 there is no error. In 8.1 i still have the PDO error. Probably that yaml stuff. I guess as long as i don't use yaml for the migrations, it'll be fine.

Thanks for your patience and all the hard work. ?

[bernhard]

18 hours ago, BlindPenguin said:

#0 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(1124): Spyc->checkKeysInValue()
#1 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(609): Spyc->returnKeyValuePair()
#2 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(545): Spyc->_parseLine()
#3 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(494): Spyc->loadWithSource()
#4 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/spyc/Spyc.php(165): Spyc->_load()
#5 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/YAML.php(11): Spyc::YAMLLoad()
#6 /home/ferdl/PhpstormProjects/process/test-project/site/modules/RockMigrations/RockMigrations.module.php(3025): RockMigrations\YAML->load()

Yes, this shows that it's related to Spyc. Seems that the library did not have any updates since 2019 so there'll likely be issues with PHP8.1 - if anybody knows better solutions please let me know.

 

[BlindPenguin]

@bernhard Symfony has a component for that. https://symfony.com/doc/current/components/yaml.html

I think it's the only one that is well supported and probably won't die anytime soon. Their components can usually be used alone too. In form of composer packages, that is.