Publish local structure changes to production without messing content

[heldercervantes]

This was probably discussed and maybe I'm just searching wrong, so apologies if I'm spamming.

In a scenario where we have a production website with a ton of content, and a local dev version with less stuff for tests and sanity. We make structural changes like adding a new page template on the local, adding a couple of fields in an existing one, changing settings on a field and whatnot, and in the end want to publish those changes to production. Any good approaches for doing such a thing without moving  the whole content around or breaking it? I mean, personally I've been doing that with DB backups and I have to lock my client out of the CMS while I'm working on updates. No likey.

Thanks,
H

[wbmnfktr]

There have been plenty of discussions and ideas about this very exact topic.

 

Right now... manual field, template and page exports could be a way to go - I don't really like it but it works most of the time.
RepeaterMatrix and some other fields are complicated or just can't be moved that way.
You have to ship around some obstacles here and there. Maybe just give it a try.

Then another viable option could be RockMigrations. It has kind of a steep learning curve but might be the swiss (or austrian in this case) army knife for migrations.

[Pixrael]

I know your feelings now, I've been there. In my experience PW is great for creation, but very bad for maintenance. The fact that the entire site configuration is dynamic based on the database tables is a big deal. One piece of advice, if you are going to use the Export/Import pages option, never think of using the ID of the pages in functions and logic in general.

[dotnetic]

Try to look into RockMigrations. It is great and makes updating fields and templates easy and version controllable.

[MarkE]

On 7/13/2022 at 7:05 PM, Pixrael said:

One piece of advice, if you are going to use the Export/Import pages option, never think of using the ID of the pages in functions and logic in general.

Very good advice. You might also like to try my ProcessDbMigrate if you don't want to go the RockMigrations route. However it does struggle with some fields - like nested repeaters. I have temporarily paused developing it while I review the best way forward, but you are welcome to try it and I will try and provide support for any problems. See the support thread here and the github here.

[bernhard]

On 7/13/2022 at 6:45 PM, wbmnfktr said:

Then another viable option could be RockMigrations. It has kind of a steep learning curve but might be the swiss (or austrian in this case) army knife for migrations.

A lot has changed since RockMigrations1. The new version is a lot easier to use! https://github.com/baumrock/RockMigrations

On 7/13/2022 at 6:01 PM, heldercervantes said:

In a scenario where we have a production website with a ton of content, and a local dev version with less stuff for tests and sanity. We make structural changes like adding a new page template on the local, adding a couple of fields in an existing one, changing settings on a field and whatnot, and in the end want to publish those changes to production. Any good approaches for doing such a thing without moving  the whole content around or breaking it? I mean, personally I've been doing that with DB backups and I have to lock my client out of the CMS while I'm working on updates. No likey.

RockMigrations was built for exactly that use case. I started in 04/2019 - so the module and its concepts have been developed, evaluated and improved for 3 years now and I'm using it in production for all of my websites.

<?php namespace ProcessWire;
/** @var RockMigrations $rm */
$rm = $this->wire->modules->get('RockMigrations');

// We make structural changes like adding a new page template on the local, 
$rm->createTemplate("your_new_template");

// adding a couple of fields in an existing one, 
$rm->setTemplateData("your_existing_template", [
  'fields' => [
    'title',
    'body',
    'add_existing_field1',
    'add_existing_field2',
  ],
]);

// changing settings on a field and whatnot, 
$rm->setFieldData('your_existing_field', [
  'label' => 'Your new field label',
  'columnWidth' => 50,
]);

Steep learning curve? I don't know... You have IntelliSense and it comes with migration snippets for VSCode and it comes with code inspector to copy and paste the syntax:

On 7/13/2022 at 6:01 PM, heldercervantes said:

, and in the end want to publish those changes to production.

# on local
git push

# on remote either login as superuser and reload the page
# or for automated deployments just use the CLI:
php /path/to/your/site/modules/RockMigrations/migrate.php

 

[heldercervantes]

@bernhard That looks super cool.

I'm now working for a studio that uses Craft CMS and although PW still feels way better in most aspects, in this one Craft has a very good solution. More or less like an automatic RockMigrations.

As you create fields, sections (templates) and whatnot, Craft generates and updates a bunch of yaml files that contain all configurations. If these files are updated, i.e. you pull someone else's changes from the repo, run a command and it applies all the changes for you. Awesome solution.

[bernhard]

3 hours ago, heldercervantes said:

As you create fields, sections (templates) and whatnot, Craft generates and updates a bunch of yaml files that contain all configurations. If these files are updated, i.e. you pull someone else's changes from the repo, run a command and it applies all the changes for you. Awesome solution.

That sounds awesome, but I can't really believe that it is as awesome as it sounds. For example, how would you create a module for craft that does the heavy lifting, but still gives you the freedom to apply changes.

An example. Note that I'm using ProcessWire language and concepts as I don't know the equivalents in craft: What if you wanted to provide a blogging module. Let's say a blog post consists of a headline, a body text and a gallery. How would you do that in craft?

Create all the necessary fields and pages via GUI and then push the yaml files to your repo? And then someone installs your module and has all those pages/templates/fields? And then what if this person wanted to add another field to the "blogitem" template? For example a date field, or an URL or an author (page reference field). How would that work in craft?

What if he added all those things via GUI? Would that break the update possibility of the blog module? Or would an update overwrite those changes?

[heldercervantes]

17 minutes ago, bernhard said:

Let's say a blog post consists of a headline, a body text and a gallery. How would you do that in craft?

That bit is very similar to PW. They have fields and templates just like us (templates are called sections). It's a bit more friendly and straightforward in PW though. They have "entries" that aren't tied to the page tree and some global stuff, but the basic way of doing things is pretty much the same. And as I create fields and sections I see these YAML files being updated to reflect the definition that's on the database.

I haven't interacted with applying changes, but I assume in a case like you described the process would somehow detect module requirements and ask you to install or update them first. I guess it could never be a miraculous solution that just does everything and works, but once a project is set up and you get to a point where devs make small increments to the site, this sounds like it will make life a lot easier.

If you're curious I found some details here. Interesting stuff.

[bernhard]

8 minutes ago, heldercervantes said:

I haven't interacted with applying changes, but I assume in a case like you described the process would somehow detect module requirements and ask you to install or update them first. I guess it could never be a miraculous solution that just does everything and works, but once a project is set up and you get to a point where devs make small increments to the site, this sounds like it will make life a lot easier.

If you're curious I found some details here. Interesting stuff.

That was the important part of my questions ? 

My point is that it would be very easy to do something similar in the PW world, where a YAML-based config is replicated to the database. But in my opinion that's useless...

I just got a new idea for RockMigrations, thx ? 

[dotnetic]

21 hours ago, bernhard said:

I just got a new idea for RockMigrations

Would love to hear that idea

[bernhard]

The idea was that RockMigrations could save at least module config settings to yaml files. I'm still not a fan of putting migrations in general into yaml files, because the execution order of migrations can matter and that's not possible to do in yaml while it is possible to do in PHP. But for module configuration I guess YAML would be an easy thing. So one could for example could install tracydebugger, do all the necessary config and hit save. Then all developers working on the project could have the same debugger settings. That might be a good thing (for module settings that need to be consistent across all environments), but it might be something you don't want. Again that's a problem that is not easy to solve and that's the reason why I think it makes the most sense to write migrations by hand.

[pwired]

Quote

This was probably discussed ...

I remember it was. Lostkobrakai posted about this with solutions. I dont have any links at the moment but will post them later on.

[dotnetic]

1 hour ago, bernhard said:

uld save at least module config settings to yaml files

Love that idea

[szabesz]

6 hours ago, bernhard said:

for module settings that need to be consistent across all environments

Something like this is already supported by @adrian's Module Settings Import / Export module which I recommend.

[heldercervantes]

How Craft warns you of changes to the YAML files that you haven't applied yet (pulled from the repo):

(I'm so used to the simplification of PW's tree that just seeing a "Routes" button in settings gives me the chills ?)

 

And this is the apply changes screen:

 

First chance I get I'm going to have a serious look at RockMigrations. Every new project I get is increasingly demanding on maintenance, with multiple devs, and this is a really big deal. This and SPAs, but thats a different discussion.

[bernhard]

Thx @heldercervantes that's interesting to see! What does the migrations menu item do or look like?

[heldercervantes]

On 7/26/2022 at 11:51 AM, bernhard said:

Thx @heldercervantes that's interesting to see! What does the migrations menu item do or look like?

Sorry, been away and lost your message.

On this particular project I don't see anything there, just a message saying "No content migrations". But it doesn't seem to be related to any of this.

I found some documentation here:
https://craftcms.com/docs/4.x/extend/migrations.html

[heldercervantes]

@bernhard coming back to this as I just stumbled upon a scenario that was discussed here.

In the Craft project I'm working on someone added changes that I needed to add to my setup. Got the message, tried applying and it stopped saying that these changes require the "Navigation" plugin. I added it (just a 'composer install' command line), came back, ran it again and done. The whole process feels very polished.

I also had conflicts in the yaml files yesterday, two devs adding new fields. The way to fix it was to include both lines, and right at the beginning there's a timestamp that Craft uses to verify if it's up to date. Used the highest one and added 1 at the end to make sure everyone got an update check.

It's really interesting to see how these yaml files are set. There's a main one that describes the project, with plugins, system stuff, some names and keys, and then there are individual files for each field, sections (templates), globals and other stuff that doesn't exist on PW.

[bernhard]

In PW using RockMigrations the developer would just have to add this to a migrations file:

$rm->installModule('Navigation');

Then the other dev could do a git pull and on the next reload everything would just work.

1 hour ago, heldercervantes said:

It's really interesting to see how these yaml files are set. There's a main one that describes the project, with plugins, system stuff, some names and keys, and then there are individual files for each field, sections (templates), globals and other stuff that doesn't exist on PW.

I'm using a similar approach for my projects. I'm having a main module for the project, eg Site.module.php that is available for all kinds of project related stuff as $site API variable. There I put the global migrations, eg this:

$rm->installModule('Navigation');
$rm->installModule('RockMatrix');

Then RockMatrix for example does also have migration files that create fields and templates that are only necessary for RockMatrix.

Works absolutely great.