RockMigrations1 - Easy migrations from dev/staging to live server

[MarkE]

4 hours ago, Richard Jedlička said:

Did you explored the Import/Export code (for fields, templates, ...) already built in ProcessWire? It is similar to what RockMigrations do (in declarative way) or not? I did not explored it much, but I am thinking if your module is needed, if there are already similar functions in a core code.

 

1 hour ago, bernhard said:

I'm not sure. I've evaluated it more than 2 years ago and never ever used it again. It was simply useless for my workflows. I've just had a glance in the core and it does not seem to be very helpful for RM.

Import/Export has a number of problems and fails to deal with some field types properly. Not wishing to detract from RockMigrations, which is excellent, but there is also my module ProcessDbMigrate. It is different in concept in that it is UI-based  rather than code-based. It does handle 'uninstallation' of migrations and also database comparisons, but there are some features of RockMigrations that it does not have. In it I used some code from Import/Export but ditched or fixed quite a bit.

[Richard Jedlička]

3 hours ago, bernhard said:

Adding data via migrations is the most straightforward thing, but removing or renaming is a little more tricky, as you realized ? Usually all methods are built in a way that they can run as often as you want. That means you can just run $rm->renameField(...) as often as you want: https://github.com/BernhardBaumrock/RockMigrations/blob/dc9dba1b050469ea085af9dca1201746e2422960/RockMigrations.module.php#L942-L944

I know that adds a little overhead and I've heard from @dotnetic that some of his migrations needed around 20 seconds to finish. On all of my setups (and I'm using RM all the time, everywhere, extensively) site-wide migrations fired via $rm->fireOnRefresh(...) take at most around 2 seconds to finish.

So you think I just have to make the code independent of how many time it ran? My reasoning wan't about it is time consuming but about it will do some changes which cannot be run more time, like some data migrations, which means I have somehow determine I was done before.

 

[Richard Jedlička]

3 hours ago, bernhard said:

I've just addes support for your request and added an example to the readme:

Great, thank you! ?

[bernhard]

6 minutes ago, Richard Jedlička said:

My reasoning wan't about it is time consuming but about it will do some changes which cannot be run more time, like some data migrations, which means I have somehow determine I was done before.

When I started with RockMigrations I had a setup similar to other migration modules where every migration was its own file and had an upgrade() and downgrade() method. I also had methods to check whether a migration should run or not (version compare). But that was a pain to work with, so over time the declarative syntax won. I agree that sometimes it would be nice to have the option to run migrations only for some versions of the project, but I have not had a good idea how to achieve that yet...

[Richard Jedlička]

1 hour ago, bernhard said:

When I started with RockMigrations I had a setup similar to other migration modules where every migration was its own file and had an upgrade() and downgrade() method. I also had methods to check whether a migration should run or not (version compare). But that was a pain to work with, so over time the declarative syntax won. I agree that sometimes it would be nice to have the option to run migrations only for some versions of the project, but I have not had a good idea how to achieve that yet...

Ok, I will try to live with that and in case of necessity I will try to find a solution.

[Richard Jedlička]

@bernhard I found a bug in your RM update, this line https://github.com/BernhardBaumrock/RockMigrations/blob/master/RockMigrations.module.php#L1007 should be

$data[$key] = $addFields;

Data under repeaterFields option should be field ids, not names, obviously. Otherwise style and script assets are not loaded for inputfields in repeater (if not present in other places in the form). It took me quite a long time to figure out that ?.

[bernhard]

Thx, fixed in v0.0.73

[Richard Jedlička]

@bernhard I've stumbled upon a problem when I want to specify more pages with the same name but different parents declaratively. More info in PR: https://github.com/BernhardBaumrock/RockMigrations/pull/23 

[bernhard]

v0.0.75 adds a new method addLanguage() that makes it super easy to add a new language to your pw site:

// get german translations for default language
$rm->setTranslationsToLanguage("https://github.com/jmartsch/pw-lang-de/archive/refs/heads/main.zip");
// add english as second language (including language field tabs and page names support)
$rm->addLanguage("en", "Englisch");

https://github.com/BernhardBaumrock/RockMigrations/commit/3e52805b9f6f3582f63826d179c753f1ad6c0750

I had a quite hard time to get this working because setting the "name" of the root page was not as easy as expected. I had to hack it via custom sql: https://github.com/BernhardBaumrock/RockMigrations/blob/3e52805b9f6f3582f63826d179c753f1ad6c0750/RockMigrations.module.php#L2235-L2242

If anybody finds a better way of doing this please let me know ? 

[dotnetic]

19 hours ago, bernhard said:

v0.0.75 adds a new method addLanguage() that makes it super easy to add a new language to your pw site:

This is sooo cool. Thanks Bernhard

[benbyf]

Hi! Is there a referrence doc for this module, love to know what i can and cant do with it and also see what need adding to pitch in... (I've looked at the github page but I can only see examples of use). I'm mostly interested in high level api's included–e.g. pages, page, fields etc (I would love to know if you can also use it for module install and settings too for example).

[Richard Jedlička]

Hi @bernhard, yes I agree with @benbyf to have some API documentation. I'm now confused with updateModuleConfig and setModuleConfig. What is the difference?

[bernhard]

Hi @benbyf

please use your IDE for that:

1 hour ago, Richard Jedlička said:

updateModuleConfig and setModuleConfig. What is the difference?

I don't know. I guess updateModuleConfig is deprecated ? 

[benbyf]

ok, thanks @bernhard...but... three things I guess, and two of them may sound stupid, so here goes..

1.  I work on a remote server so this simply won't work for me (as far as i can tell)... on small sites i tend to have a dev site on my dev server and role out to live when its done or close too, my IDE just sees the files I'm editing.
2. I seem to be drawn to the array/json esque way of using the migrate function instead of each individual function, so again IDe wont help here i don't think?
3. Less stupid I hope but, my IDE doesn't give me YOUR preferred usage, things left TODO / roadmap so we can pitch in, or other ways to interact with the module... which in this case seem like there are lots of usecases and places to use it.

 

[bernhard]

10 minutes ago, benbyf said:

 I work on a remote server so this simply won't work for me (as far as i can tell)... on small sites i tend to have a dev site on my dev server and role out to live when its done or close too, my IDE just sees the files I'm editing.

If your code completion does not work properly you are making your life unnecessarily hard and you should definitely find a way to make that work. It's like moving around in a dark room and then finding the switch to turn on the light. If you like darkness better docs for RockMigrations won't help ? 

13 minutes ago, benbyf said:

I seem to be drawn to the array/json esque way of using the migrate function instead of each individual function, so again IDe wont help here i don't think?

Array syntax just uses all the properties that PW uses and you can see those properties using TracyDebugger... Another switch to turn on the light ? 

14 minutes ago, benbyf said:

Less stupid I hope but, my IDE doesn't give me YOUR preferred usage, things left TODO / roadmap so we can pitch in, or other ways to interact with the module... which in this case seem like there are lots of usecases and places to use it.

I don't understand what you are saying and neither does deepl, sorry. Note that not everybody here is a native speaker..

[benbyf]

sooooo, how do you develop with PW then? Love to learn more about different routes (used things like MAMP and even Vagrant in the past but felt less friction the better so moved to a remote only way of working)

Last bit refers to how other devs can get involved with the git project.

[bernhard]

I'm developing locally on my laptop using VSCode and Intelephense and I'm using RockMigrations to replicate everything that I did locally to the remote server. That's all it is about afterall, so I'm not sure why one would use RockMigrations if he/she did all the changes remotely anyhow?! No need for migrations then...

 

[benbyf]

Fair. I like the idea of starting projects by configuring in code rather than in the admin. Also means i have the option of using it for migrations in future between dev and live.

[dragan]

21 hours ago, benbyf said:

my IDE doesn't give me (...) things left TODO / roadmap

My IDE does ?

PHPStorm has a tab that checks code comments starting with @todo - VSCode has probably something similar as a plugin I guess.

[adrian]

@benbyf - you might prefer Tracy's API Explorer for getting the docs:

[benbyf]

On 12/24/2021 at 6:01 PM, dragan said:

My IDE does ?

PHPStorm has a tab that checks code comments starting with @todo - VSCode has probably something similar as a plugin I guess.

Good point, I've just had a look in VScode and theres a few.

On 12/26/2021 at 11:18 PM, adrian said:

@benbyf - you might prefer Tracy's API Explorer for getting the docs:

Will take a look ?

[bernhard]

Several updates have been pushed to the module. The latest one concerns the fireOnRefresh method / strategy. If you are using this technique together with bootstrapping ProcessWire be sure to test everything as this change might introduce different behaviour than before! From the readme:

Note about fireOnRefresh()

Prior to v0.0.82 the fireOnRefresh did NOT fire in bootstrapped environments. This is because ProcessWire by default runs as guest user when bootstrapped and the fireOnRefresh method did not attach any hooks if the user was no superuser. That was a problem for me because a $modules->refresh() did not trigger any migrations when invoked from the commandline. To solve that issue v0.0.82 introduces another concept:

fireOnRefresh fires always after Modules::refresh even for guest users. If you need to prevent RockMigrations from firing actions that where attached via fireOnRefresh you have two options:

• Setting a constant: define("DontFireOnRefresh", true)
• Setting a config property: $config->DontFireOnRefresh = true;

Note that the setting must be available to RockMigrations BEFORE the action gets attached via fireOnRefresh. This is best explained by examples:

define('DontFireOnRefresh', true);
include("index.php"); // fireOnRefresh is triggered here
$modules->refresh(); // this will not trigger any migrations

We define the constant before ProcessWire gets loaded and therefore any module attaching migrations via fireOnRefresh will actually NOT attach any migrations because the flag to prevent that is present when the triggered from init() or ready().

include("index.php"); // fireOnRefresh is triggered here
$config->DontFireOnRefresh = true;
$modules->refresh(); // this WILL trigger all migrations!

In this example the migrations will be triggered because the migrations have been attached BEFORE the config setting was set. In other words the flag was set too late for RockMigrations to realize it.

[gebeer]

Hi @bernhard

 shows a test of recording changes in templates/fields to yaml. This looks very promising. Do you have any plans integrating this into your module?

It is easy to use RockMigrations when you start a new project. But for existing projects where RM comes in later, we need to write the migration files by hand before we can start using rm()->migrate. Would it be possible to create a yaml from all templates/fields of an existing install based on the recorder that you showed in that video?

[bernhard]

15 hours ago, gebeer said:

It is easy to use RockMigrations when you start a new project. But for existing projects where RM comes in later, we need to write the migration files by hand before we can start using rm()->migrate.

I think I don't agree with you here. Could you please explain that in detail?

15 hours ago, gebeer said:

Would it be possible to create a yaml from all templates/fields of an existing install based on the recorder that you showed in that video?

I'm not sure I understand. What you are asking for is exactly what I'm showing in the video?!

[gebeer]

On 1/16/2022 at 12:50 AM, bernhard said:

I think I don't agree with you here. Could you please explain that in detail?

I think I had a wrong understanding of the migrate([]) method, thinking it is destructive for fields/templates that are not in the $config array. So if I have an existing site, I would have to build the $config array with all fields and templates that are in the existing installation already. If I forgot one field, it would be deleted by the next call of the migrate method. But looking at the code, I see that it only creates fields and templates.
But still, if I don't add fields/templates to the $config array, that are already there in an existing installation, the migration would not cover all fields/templates. Does that make sense?

On 1/16/2022 at 12:50 AM, bernhard said:

I'm not sure I understand. What you are asking for is exactly what I'm showing in the video?!

Yes, this is exactly what happens in the video. All fields/templates of a site are getting written to the yaml file. It would be great, if we had this available for setting up initial migration() $config arrays. Either inside the module or as a separate "recorder" module. That way, we could plugin RockMigrations to any existing site, create the initial $config data (inside a yaml file) and move on from there with our migrations.