RockMigrations1 - Easy migrations from dev/staging to live server

[bernhard]

Hi @zoeck

10 minutes ago, zoeck said:

And a second problem with an option field:

See setFieldOptionsString() method

11 minutes ago, zoeck said:

Are there any problems when you create repeaters and use fields that are created in the same upgrade?

There is no error, but my fields were not assigned to the repeater.

I guess I've never came up with any migration for repeater fields, so this might need some custom integration. PRs welcome. I don't use Repeaters at all these days ? 

[zoeck]

23 minutes ago, bernhard said:

Hi @zoeck

See setFieldOptionsString() method

Is this possible with the "migrate" function?

Just Added one Line on the "Upgrade" with the setFieldOptionsString -> working ? Thanks!

 

Migrations without repeaters ? thats uncool... 

Do you not use repeaters at all?

[bernhard]

11 minutes ago, zoeck said:

Is this possible with the "migrate" function?

No. migrate() is just a shortcut array syntax. You can always combine both:

$rm->migrate([
	'fields' => ['foo' => ...]
]);
$rm->setFieldOptionsString('foo', ...);

 

11 minutes ago, zoeck said:

Migrations without repeaters ? thats uncool... 

?

11 minutes ago, zoeck said:

Do you not use repeaters at all?

No. Pages for the win. A lot easier to handle in many situations (Migrations, RockFinder, ...). But I have a custom Repeater implementation... Maybe the need for migrating repeaters will arise one day for me and maybe I'll implemenet something then if nobody that needed it earlier did submit a PR...

[zoeck]

On 5/28/2020 at 2:23 PM, bernhard said:

I guess I've never came up with any migration for repeater fields, so this might need some custom integration. PRs welcome. I don't use Repeaters at all these days ? 

After I had a closer look at the repeaters, I could add the items to the repeater.
It is actually not that difficult, but you need a little workaround:

$repeatertemplate = $this->templates->get($this->fields->get("**REPEATERNAME**")->template_id);
$rm->addFieldToTemplate("proj_milestone_date",$repeatertemplate);

For each repeater a (hidden) template is created, you just have to find the template from the repeater and then add the field via RockMigrations ? 
It is also possible to add several fields with "addFieldsToTemplate", setFieldData is also possible.

[bernhard]

Thx @zoeck I've added a shortcut method getRepeaterTemplate() for that: https://github.com/BernhardBaumrock/RockMigrations/commit/dd741064cd19b661bdcb441cdabe153b3abe8e57

[zoeck]

8 minutes ago, bernhard said:

Thx @zoeck I've added a shortcut method getRepeaterTemplate() for that: https://github.com/BernhardBaumrock/RockMigrations/commit/dd741064cd19b661bdcb441cdabe153b3abe8e57

Works without problems, thanks ? 

[elabx]

On 5/19/2020 at 4:59 AM, bernhard said:

Would be great to hear your experience then. Is @LostKobrakai's module really obsolete or does it still have its place?

Just to throw an opinion here, I still use @LostKobrakai module due to the command line support and I do like the way it works and how it keeps track of migrations run, which I'm not sure it's done in RockMigrations. But I guess RockMigrations could be made to work by invoking migrations script with the php command and add some tracking, I do like a lot how the field migrations are described here. 

[bernhard]

Thx for that report @elabx ? The other module works totally different. My migrations can be run from anywhere, anytime. That makes it a lot easier to use but of course this has drawbacks. Keeping track of with migrations have already been applied is more complicated. I'm not sure how that could be done exactly. But I have to say that since I'm using RockMigrations I have not REALLY needed such a feature. My main goal was to make the changes as easy as possible. Migrations can always be applied several times so it does not really matter if they have been applied or not.

But I'd love to have a good concept for that and since RockMigrations is totally free and open source I invite anybody to contribute and improve it ? I have lots of other more important things to do.

Command line support sounds interesting though. But I'm not sure if that would bring a huge benefit compared to the tracy console. And even more I wonder if it might make a lot more sense to have a command line module that can be used by any other module. Not one that is locked to RockMigrations...

[elabx]

1 hour ago, bernhard said:

But I'd love to have a good concept for that and since RockMigrations is totally free and open source I invite anybody to contribute and improve it ? I have lots of other more important things to do.

Didn't want to imply your module needs anything or that it is "missing" something or that I was asking for a feature, I apologize if that's how it came out! Just wanted to give an example of an exception to the norm in case anyone finds this thread and of course, I agree it's the mission of all of us to contribute to open source and I must say this community has inspired me a lot to thrive into that direction rather than just consuming the work of others but also contributing, first step is to solve the very serious personal time management issues I have and just force me to do get  open source done lol.

1 hour ago, bernhard said:

Command line support sounds interesting though.

So basically how it's useful for me is that, I keep a bunch of sites on one server and they all share template files and fields configuration. I keep files in sync using git and git worktrees, and with a shell script I make merges from the dev branch into the rest of the branches, including the migration files (each site is a branch). Then, if I need to make an update that involves any adding/removing fields I also have another shell script that goes on every site directory and runs a specific migration. It saves me a lot of time to run this for the 50+ sites I manage, I can very easily see if an error occurred, if migration was already run, if it didn't exist (then it means that site is now correctly wired up to the git repo), etc. 

Probably this is the least likely scenario of most PW users lol, as I have the feeling that most of PW users build heavily customized sites, for very specific requirements. 

[bernhard]

1 hour ago, elabx said:

Didn't want to imply your module needs anything or that it is "missing" something or that I was asking for a feature, I apologize if that's how it came out! Just wanted to give an example of an exception to the norm in case anyone finds this thread and of course, I agree it's the mission of all of us to contribute to open source and I must say this community has inspired me a lot to thrive into that direction rather than just consuming the work of others but also contributing, first step is to solve the very serious personal time management issues I have and just force me to do get  open source done lol.

Didn't take that as offense at all ? Just tried to make clear that I have already thought of that but I don't plan to implement it any time soon. But if anybody would be up for it, I'd be more than happy to assist or to pull the changes ? Sorry if that was not clear.

[LostKobrakai]

12 hours ago, elabx said:

Probably this is the least likely scenario of most PW users lol, as I have the feeling that most of PW users build heavily customized sites, for very specific requirements.

I'm sure it's the less used path, but a CLI is useful for anybody automating deployments either using some kind of CI/CD or manually called scripts.

13 hours ago, bernhard said:

The other module works totally different. My migrations can be run from anywhere, anytime. That makes it a lot easier to use but of course this has drawbacks. Keeping track of with migrations have already been applied is more complicated. I'm not sure how that could be done exactly.

The goal of my migrations module was to result in reproducable db state, so automation is possible. If migrations can be run arbitrary times it's not going to do that – e.g. a field created and later deleted is not the same as a field deleted and later created. Therefore migrations have a fixed order (by timestamp) and a db table keeps knowledge about which migrations were already applied before.

[bernhard]

v0.0.11 adds support for repeater matrix fields ? 

Thx to @aComAdi of https://www.a-commerce.ch/ for sponsoring this update! ?

$rm->setMatrixItems('your_matrix_field', [
  'foo' => [
    'label' => 'foo label',
    'fields' => ['field1', 'field2'],
  ],
  'bar' => [
    'label' => 'bar label',
    'fields' => ['field1', 'field3'],
  ],
]);

 

[bernhard]

v0.0.19 adds support for RepeaterFields @zoeck

$rm->migrate([
  'fields' => [
    'my_repeater_field' => [
      'type' => 'repeater',
      'label' => 'This is my great repeater field',
      'repeaterFields' => ['title', 'body', 'images'],
    ],
  ],
]);

 

[zoeck]

55 minutes ago, bernhard said:

v0.0.19 adds support for RepeaterFields @zoeck

Thanks @bernhard ? nice Update!

[sebr]

hello

@bernhard Thanks a lot for this dreamy, powerful and simple migrations module ? It's great and I begun to use it with succes !

I have however one problem ? :

1. I created a module and I use the auto-update system from RockMigrations (0.0.1.php). All is OK with the install and the uninstall. I create a 0.0.2.php and update the version of the module ('version' => '0.0.2'). But when I refresh modules from Processwire, my module update in 0.0.2, but no RockMigrations is executed.
2. When I debug it with Tracy Debuger, I have many errors like this :

   array_key_exists(): Using array_key_exists() on objects is deprecated. Use isset() or property_exists() instead on line: 171 in /var/www/html/wire/core/WireData.php

   and another error like this one

   Maximum function nesting level of '256' reached, aborting! on line: 1733 in /var/www/html/wire/core/Wire.php

Any idea to resolve it ? I tried to increase the maximum function nesting level, but it's the same... ?

Thanks for your help

[bernhard]

9 hours ago, sebr said:

@bernhard Thanks a lot for this dreamy, powerful and simple migrations module ? It's great and I begun to use it with succes !

Hi @sebr, happy to hear that ? 

9 hours ago, sebr said:

I created a module and I use the auto-update system from RockMigrations (0.0.1.php). All is OK with the install and the uninstall. I create a 0.0.2.php and update the version of the module ('version' => '0.0.2'). But when I refresh modules from Processwire, my module update in 0.0.2, but no RockMigrations is executed.

9 hours ago, sebr said:

Any idea to resolve it ? I tried to increase the maximum function nesting level, but it's the same... ?

Unfortunately, no. And I have no time to investigate. I can only share my experience: In my setups/modules I tend to use one single migrate() method now. Instead of having a migration for every version this approach just makes sure that after running migrate() you end up with the setup you want (fields, templates, pages). This is very easy using the RM migrate() method:

$rm->migrate([
  'fields' => [
    'foo' => ['type' => 'text'],
  ],
  'templates' => [
    'foo' => ['fields' => 'foo'],
  ],
]);

Lets say in the next version I need a new field "bar", I'd just add it to that migration:

$rm->migrate([
  'fields' => [
    'foo' => ['type' => 'text', 'label' => 'Foo Label'],
    'bar' => ['type' => 'text', 'label' => 'Bar Label'],
  ],
  'templates' => [
    'foo' => ['fields' => 'foo', 'bar'],
  ],
]);

This makes it really easy to grasp the setup, because you have everything in one place and you don't need to go through every migration and try to understand the whole picture. The example above with the "old" approach would be:

$rm->setFieldData('foo', ['label' => 'Foo Label']);
$rm->createField('bar', ['label' => 'Bar Label']);
$rm->addFieldToTemplate('bar', 'foo');

Not to forget that with the new approach you can diff your changes:

Of course this comes with the drawback of a little less control and it can get tricky when multiple modules to migrations and one depends on the other. But then you can still split migrations up or do necessary changes upfront. Also removing things needs some extra lines of code, but still very easy ones:

$rm->deleteField('bar');
$rm->migrate([
  'fields' => [
    'foo' => ['type' => 'text', 'label' => 'Foo Label'],
  ],
  'templates' => [
    'foo' => ['fields' => 'foo'],
  ],
]);

On uninstall I just do all the cleanup and that's usually as easy as deleting all templates and fields (because all pages get deleted automatically).

I've never ever needed to run a downgrade method. I usually have a cleanup migration that just removes fields or templates that I created during dev and maybe even pushed to the live server and then I change something and dont need those fields any more (like the "bar" field above).

RM is still not perfect. I had a chat with horst and I'd like to have logging and test-run capability, but for now this is what we have and it is saving me a ton of time and headache already and I can really not understand how anybody can work with ProcessWire without RM ? 

[sebr]

Thanks for this response. Yes, I also think this is a good solution : just one migrate call. I my case, downgrade is not necessary. The only risk I take concerns manual changes that could be done directly under Processwire. Have a nice day !

[cjx2240]

Can this module be used to do something like, change the allowed filetypes of a file field, or change the available options in an options (select) field?

[bernhard]

Of course it can:

https://github.com/BernhardBaumrock/RockMigrations/blob/a034abce7e7fa5436756a2506d9d5f17d8a1b361/RockMigrations.module.php#L622-L638

https://github.com/BernhardBaumrock/RockMigrations/blob/a034abce7e7fa5436756a2506d9d5f17d8a1b361/RockMigrations.module.php#L725-L736

[eydun]

On 7/22/2020 at 10:54 PM, sebr said:

When I debug it with Tracy Debuger, I have many errors like this :

1. 
   array_key_exists(): Using array_key_exists() on objects is deprecated. Use isset() or property_exists() instead on line: 171 in /var/www/html/wire/core/WireData.php

   ?

I think this is related to php 7.4 has deprecated "array_key_exists". If you can downgrade to php 7.3, then that will remove this message.

[bernhard]

Thx @eydun but your error message states that this error comes from the core and not from my module ? 

[eydun]

You are correct ? But I think your excellent RockMigrations.module.php also uses the array_key_exists-function.

[eydun]

BTW, the new migrate-method together with Tracy Console is a game changer ??

[bernhard]

16 hours ago, eydun said:

You are correct ? But I think your excellent RockMigrations.module.php also uses the array_key_exists-function.

the function itself is not deprecated - just its use on objects is. I don't know if I'm using it like this anywhere. If I do, please let me know. If not - everything's fine ?

1 hour ago, eydun said:

BTW, the new migrate-method together with Tracy Console is a game changer ??

? 

[benbyf]

Am I right in saying that this module is less about migrations and more about automation? As using methods to do work using handy functions in a linear way sounds like automation to me, where as migration is something I want to do when I have a bunch of stuff that has happened to the environment that needs replication for some reason (i.e exporting and importing fomr staging to live).

I only mention it as ideally you would work on your PW install (in the admin or in code) and at certain points take a migration of the changes... and therefore not need to hand write the migrations at all, as they're actaully artifacts of changes to the db?

Bit meta, but I like using the admin, and would rather migrations were something that happened, not that needed writing. (this is coming from some experience with a migrations module in Drupal 6... yes i know! That did just as explained)