RockMigrations 🚀🚀 The Ultimate Automation and Deployment-Tool for ProcessWire

[JayGee]

30 minutes ago, bernhard said:

Hi @Guy Incognito thx for the input. RockMigrations needs PHP>=8.0 and ProcessWire should warn you about that before installation: https://github.com/baumrock/RockMigrations/blob/1089f625048c4c68e883e46d1d57d7153c155bf2/RockMigrations.info.php#L14

So you did a fresh install and got an error without PW showing the requirement fails?

Ah ok - I didn't clock the PHP 8.0 requirement apologies. I just installed on auto pilot without thinking as have used your module before 🙂. It's odd though as I thought that operator had been supported pre 8.

It wasn't a fresh PW install it was an existing project. I don't recall seeing the warning about PHP version, but Tracy Debugger intercepted the syntax error when I refreshed the modules, so I didn't see the default PW messaging.

But all running smoothly now anyway thanks.

[bernhard]

Great 🙂 

If you had RM < 3.0 be sure to check out this post: https://processwire.com/talk/topic/27504-rockmigrations-🚀🚀-the-ultimate-automation-and-deployment-tool-for-processwire/?do=findComment&comment=230651

 

[bernhard]

I've just released a new version with a ton of updates 😎 https://github.com/baumrock/RockMigrations/releases/tag/v3.25.0

Quote

v3.25.0 Latest

Bug Fixes
•    add early exit if tpl is not set (2d35674)
•    livereload preventing module installation (db4d18f)
•    remove transition that has no effect (e6524ac)

Features
•    add $rm->inputfield() helper method (c548a78)
•    add backend stylesheet (b45bbf2)
•    add echo() method for cli usage (f09dd3c)
•    add email field (7418757)
•    add getTplName() method to magicpages (bfe4fab)
•    add onChanged() magic method (04ffe16)
•    add pageclass snippet (f2be261)
•    add pageListBadge() helper (8b202ca)
•    add PageListQuickActions tweak (fa8f857)
•    add PageListShowTemplate tweak (40c49e8)
•    add permissions- and access- keys for migrate() (890c242)
•    add removeSaveButton() to magicpages (fc1e56f)
•    add removeSubmitActions() (25701f9)
•    add rm-colorbar class to colorbar (5d56b2b)
•    add rootVersion() (3ab7560)
•    add support to remove all submit actions (5c39faa)
•    add title field to migrated pageclasses (c5bd95a)
•    improve deployment and its logs (7120e5c)
•    improve labels (64cf73e)
•    show deployment variables in log (41bc4e7)
•    update docs and rename deploy workflow (5bd96f3)

I started to adopt a new workflow, because I often heard that "it's hard to keep track". That's why from now on I'll push updates to the dev branch and then merge them to main from time to time.

Highlights:

I improved the deployment log a lot, made everything a lot cleaner and tried to catch all the warnings that where just written to the log before.

Old:

New:

We also have a new tweak to show page ids in the page tree (1) and a helper method to add labels in the page list (2) and also the template name (not in the screenshot)

Check out https://github.com/baumrock/RockMigrations/releases/tag/v3.25.0 for all updates!

[bernhard]

New month, new version: https://github.com/baumrock/RockMigrations/releases/tag/v3.26.0

Bug Fixes
double linebreaks (bf07c6b)
magicpages comparing classnames without namespaces (28b5657)
prevent migrating pageclasses twice (b3c9add)
remove double slash when using $this->path (aa9502c)
wirerandom issue on deployment (29bd145)

Features
add placeBefore to wrapFields() (906753c)
add random salts to config-local suggestion (1e7401c)
add rm-hints to field and template gui (27b5b25)
add support for parent_id as page path (e45e430)
imporve createPage for array syntax (74bd338)
improve getTemplate (use ::tpl fallback) (b6f6d14)
improve isDDEV() (e4df541)
improve setFieldData() method (2830eed)
use $this->path instead of DIR #22 (f134e1b)

One very nice feature is that RockMigrations now adds uikit tooltips to the field and template editor where you can directly inspect the name of the property to use and also the value of one setting, which is very handy if you need a setting that you don't know by heart. With this update you don't need to find the setting by inspecting the html markup any more and you can instantly see it by hovering over the setting:

[snck]

I am using RockMigrations for a while now (and am honestly in love with it!) and just recently ran into a problem that I had not encountered before. I am using the deployment features and a setup that automatically deploys the two (dev and main) branches in my Github repo to different locations. This works perfectly fine most of the time, however last time PHP ran out of memory and I did not notice it, because the job was marked as successful in Github actions. Running out of memory led to the current symlink not getting generated and the website being not accessible (without me noticing it).

@bernhard and I already had a chat about it, but we were wondering whether you guys had ideas or suggestions on how to deal with errors and warnings during the deployment process and appreciate your input.

[gornycreative]

On 3/24/2023 at 6:40 AM, bernhard said:

I only place the fields on the correct template, which is necessary for the fields to be queryable by selectors etc and all the rest is done in code in the pageclass.

I'm looking at the code for wrapFields() - is this only for Fieldset and not FieldsetTab?

If you are using custom tabs is adding a FieldsetTabOpen field type still recommended and then

$rm->addFieldToTemplate('fieldname', 'content-page' , 'tab', 'tab_END');

still the best way to do it?

[bernhard]

1 hour ago, gornycreative said:

If you are using custom tabs is adding a FieldsetTabOpen field type still recommended and then

$rm->addFieldToTemplate('fieldname', 'content-page' , 'tab', 'tab_END');

still the best way to do it?

Hey @gornycreative thx for the question!

This is something that I'd like to have but still haven't found time to implement. Maybe you could add a PR?

I'm using wrapFields all the time and I'd love to have a setting to make it wrap fields in a tab 🙂 

[gornycreative]

Okay.

I'm playing around a little bit with the getCode functions to pull names instead of IDs for certain things and looking at the repeatermatrix situation a little more but I will also look at the tab add situation after.

[PavelRadvan]

Hi.
Nice module.
Is possible to use this module to import data to PW ?
As it is named "Migrations" it could migrate data (from xml/csv...or other) to PW - could it be done with it?

 

[bernhard]

Sure. Why should that not be possible? But there are no GUI import/export tools or the like. But you can write your scripts and use everything that RockMigrations provides. Though maybe you wouldn't even need RockMigrations for that... Just use the PW API

[PavelRadvan]

7 hours ago, bernhard said:

Sure. Why should that not be possible? But there are no GUI import/export tools or the like. But you can write your scripts and use everything that RockMigrations provides. Though maybe you wouldn't even need RockMigrations for that... Just use the PW API

Thanks for quick answer.
I would like to know what you could recommend for my current project.
For right answer I have to explain situation:
I am working on customer request to have some app for managing his business contracts and assets.
His is doing business with antiquities - so he is receiving valuable items from customers and offers them for sale.
Mainly these items are jewellery, art and other antiquities (lots of them).
Now he is using very old Windows app based on very old MS Access db.
This old app has db with all of his current and old contracts inf form of interconnected tables and also links to tons of images of items (antiquities).
So for one contract there could be several connected images. To this day there are lots of images (around 50 000 jpeg/png..) and lots of data in app.
This app is not good now for his needs. It takes very long to search anything and also working with images and data is not good..
I decided that I could use PW for his app.
And now my questions:
Is better to use PW with your modules for import data from that Windows app or is better to use some import module like ImportPagesCSV, link: https://processwire.com/modules/table-csv-import-export/ ?
Which other modules would you recommend ?
Thanks for your answer.
Pavel Radvan

[dotnetic]

Hello @PavelRadvan. I don't know how much experience with ProcessWire you have, and why you chose it. You can definetly develop an application to search through the database, but as much as I love ProcessWire, I think there are better tools for this task.

First you have to export the data from Access to a MySQL or MariaDB.

Then I would recommend using a tool like Directus (free, open source), with which you could Add Directus to an Existing Database (learndirectus.com)

Or Filament (free, open source) which is a nice admin for databases and CRUD functionality.

Or Laravel Nova.

[bernhard]

I disagree, but I don't know the tools you mentioned @dotnetic

But I don't see any reason why PW would not be a great fit for such a project. In general I think that PW is a brilliant replacement for MS Access databases. Only thing missing in my opinion is RockGrid for displaying data in an excel-like manner with sorting and filtering features.

12 hours ago, PavelRadvan said:

Is better to use PW with your modules for import data from that Windows app or is better to use some import module like ImportPagesCSV,

This has nothing to do with my modules. RockMigrations has a different purpose. It's not about migrating data from other systems to PW it is about migrating changes that you added on your development system to staging or production systems.

So for example if you were done with the move from MS Access to PW and the client was already working on your system and then one day he/she comes to you and says: "nice, everything works great, but we want another feature XY!".

You know that this feature will take 3 weeks to develop. What would you do?

One option would be to take the live site offline for that 3 weeks and then, when you are done, upload everything to the live server. That's usually not a good solution.

The other option is to leave the site online and develop everything locally in parallel using the power of RockMigrations. All the database changes (creating templates, fields, roles, etc) can be done in code and that means once you are done you can just upload your changes to the live server, migrations will kick in and create all the changes that are necessary and your system will work just like it did on your local development and your client will get the update with zero downtime.

That answers the question related to RockMigrations I guess. Everything else is offtopic, so please either create a new topic and ask the community for help or write me a PM if you want to purchase consulting from me personally. Thx.

All the best for your project. It sounds interesting and fun 🙂 

[PavelRadvan]

1 hour ago, dotnetic said:

Hello @PavelRadvan. I don't know how much experience with ProcessWire you have, and why you chose it. You can definetly develop an application to search through the database, but as much as I love ProcessWire, I think there are better tools for this task.

First you have to export the data from Access to a MySQL or MariaDB.

Then I would recommend using a tool like Directus (free, open source), with which you could Add Directus to an Existing Database (learndirectus.com)

Or Filament (free, open source) which is a nice admin for databases and CRUD functionality.

Or Laravel Nova.

Hi.
Thanks very much for your answer. Tools your writing are interesting, but application I want is not only for searching in database. I do not have vast experience with PW, but I have several sites built with it and I have to say it is wonderfull system.
For me it is decided selection.

[PavelRadvan]

1 hour ago, bernhard said:

I disagree, but I don't know the tools you mentioned @dotnetic

But I don't see any reason why PW would not be a great fit for such a project. In general I think that PW is a brilliant replacement for MS Access databases. Only thing missing in my opinion is RockGrid for displaying data in an excel-like manner with sorting and filtering features.

This has nothing to do with my modules. RockMigrations has a different purpose. It's not about migrating data from other systems to PW it is about migrating changes that you added on your development system to staging or production systems.

So for example if you were done with the move from MS Access to PW and the client was already working on your system and then one day he/she comes to you and says: "nice, everything works great, but we want another feature XY!".

You know that this feature will take 3 weeks to develop. What would you do?

One option would be to take the live site offline for that 3 weeks and then, when you are done, upload everything to the live server. That's usually not a good solution.

The other option is to leave the site online and develop everything locally in parallel using the power of RockMigrations. All the database changes (creating templates, fields, roles, etc) can be done in code and that means once you are done you can just upload your changes to the live server, migrations will kick in and create all the changes that are necessary and your system will work just like it did on your local development and your client will get the update with zero downtime.

That answers the question related to RockMigrations I guess. Everything else is offtopic, so please either create a new topic and ask the community for help or write me a PM if you want to purchase consulting from me personally. Thx.

All the best for your project. It sounds interesting and fun 🙂 

Thanks for your answer and explanation. I certainly will use PW and now I clearly know that your modules are also very usable in my project.

[bernhard]

Monthly update: RockMigrations v3.27.0 is out 🙂 

Quote

Bug Fixes

• createUser returning false PR#24 (fb18156)

Features

• add deployment hooks (fcefd1f)
• add PageListAutoExpand to default profile (68af9f9)
• add rockhsell demo command (cce0d06)
• add support for getTplName() syntax (c01630f)
• add time constants (17ab452)
• make github action fail on error in migration (7499223)

 

Deployment hooks

You can now hook into any stage of your deployment and apply any custom commands easily. For example this hook would remove the .htaccess file and rename the .htaccess-staging to .htaccess:

$deploy->after("share", function ($deploy) {
  $release = $deploy->paths->release;
  $deploy->exec("rm $release/.htaccess");
  $deploy->exec("mv $release/.htaccess-staging $release/.htaccess");
});

 

Time constants

RockMigrations now comes with some helper constants that are easier to read: 

  // time constants (seconds)
  // see https://i.imgur.com/vfTasHa.png
  const oneMinute = 60;
  const oneHour = self::oneMinute * 60;
  const oneDay = self::oneHour * 24;
  const oneWeek = self::oneDay * 7;
  const oneMonth = self::oneDay * 30;
  const oneYear = self::oneDay * 365;

 

RockShell demo command

RockShell has been updated to support shipping PW modules with RockShell commands. Just create a file /RockShell/Commands/YourCommand.php and that command will be available in your RockShell CLI. RockMigrations now comes with an example command to get started.

[gornycreative]

I don't know why, but I'm not able to get watched files to trigger. They appear in the watchlist when I run a dump. There are rockmigrations methods in the files.

I've set the files to force = true in the watch method in migrate.php. I'm at my wits end.

migrate.php triggers on change. Page classes trigger on change. I can't get a single watchfile to trigger. Is there a format to watchfiles that I am missing?

Edit: The reason I am running into this now is because I had been keeping field migrate definitions and matrixtype definition methods all on one file but that is starting to get hefty and so I wanted to break out the matrix definition types for various repeatermatrix fields into separate files that would only trigger when a new type was added.

[bernhard]

Hi @gornycreative it would be nice to create a new topic with such requests so it's easier to find these topics, link to them and not get multiple discussions in parallel in one thread.

To your problem:

"Watch" does only mean that the file get's watched for changes. If rockmigrations recognises a change, then migrations will get triggered. "Force" means, that migrations will get triggered on every run. This can be necessary when you have a layered setup, for example you have one module that manages several sub-components, like for example RockPageBuilder manages all migrations for all Blocks. There we need to run migrations of RockPageBuilder all the time to make sure all blocks have the correct foundation. This explanation is not 100% accurate I think but you get the idea.

In your case you could either break your migrations in part in different methods. I'd probably place them in my site module, as that's something project-specific. In that site module (/site/modules/Site/Site.module.php) you could have one migrate method:

<?php
public function migrate() {
  $this->foo();
  $this->bar();
  $this->baz();
}

public function foo() {
  $rm->createField(...);
}

...

Your approach should also work, but I guess it depends on the content of your split files. I think you can't just put $rm->something() into that files as you can do in /site/migrate.php - that will only work in /site/migrate.php and I think also in /site/modules/Foo/Foo.migrate.php

For module files for example RM needs to load the module first and then trigger the migrate() method. Other files need to be loaded directly (like /site/migrate.php). Maybe you want to share one of your split migration files and we can have a look?

[bernhard]

Lots of great stuff in RockMigrations v3.29.0

Spoiler

v3.29.0 Latest

 

 

Bug Fixes

• catch 404 page not found error on deployment (e67bebd)
• jquery deprecation warning for window.load() (c626d5b)
• pageListBadge showing when empty (3644e6a)
• remove language tab fix (4a1061b)
• renderTable not showing data and wrong asset path on windows (24a9d9d)
• renderTable not showing some values (bce2df8)

Features

• add checkbox to force livereload on module config pages (a52a550)
• add getFile() method (ea94430)
• add homeTemplate() method (cbd0c07)
• add new pageClassLoader 🙂 (480b06c)
• add path helper method (29e8740)
• add redirect logger Tweak (6003cf5)
• add renderTable helper (eb3ee9c)
• add support for template_ids (14489ad)
• improve new pageClassLoader (e99b80c)

Sorry for the ugly background but otherwise the links don't work!

[jploch]

I am thinking about integrating RockMigrations in my workflow, but would like to make sure I understand it correctly before. 
On most of my projects I am the only one developing the website. I usually develop locally and push changes to a production server.
So I FTP the files and either use the core native field/template export/import in the GUI or make the changes manually in the GUI.

For more complex things, I also sometimes write my own litte scripts to automate stuff using the PW API and put them in the site/ready.php or a seperate module.
I have helper functions to simplify the template and field creation. The benefit of RockMigrations seems to be to simplifying/abstracting the native PW API and helping with triggering the migration? I can see the benefit of it but I am not sure if I want to depend on RockMigrations for this, since the native API is already quite easy to use.

An automation I wrote using native API and helper functions might look like this:

// function to create template and fields
// you could run this function on module resfresh for example
  public function create() {

    //install dependencies
    if (!$this->modules->isInstalled('InputfieldTinyMCE')) {
      $this->modules->install('InputfieldTinyMCE');
      $this->modules->refresh();
    }

	//call to helper functions
	//the functions will make sure it only creates the template and field if they don't exist
   	$this->createTemplate('example-template');
    $f = $this->createField('example-field', 'FieldtypeTextarea', 'example-template');

    $f->inputfieldClass = 'InputfieldTinyMCE';
    $f->label = 'Text';
    $f->contentType = 1;
    $f->features = array(0 => 'toolbar', 1 => 'stickybars', 2 => 'spellcheck', 3 => 'purifier', 4 => 'pasteFilter');
    $f->save();
  }

[bernhard]

2 minutes ago, jploch said:

The benefit of RockMigrations seems to be to simplifying/abstracting the native PW API and helping with triggering the migration? I can see the benefit of it but I am not sure if I want to depend on RockMigrations for this, since the native API is already quite easy to use.

RockMigrations has a lot more to offer. It has a proven workflow for professional and automated deployments for example and it comes with MagicPages that have a lot of great helpers if you are using custom page classes. It has helpers to make working with the page editor easier (eg the wrapFields() method) and it has helpers to quickly see all the properties you need via mouse hover.

Even if I died tomorrow you could just continue using RockMigrations for the next 10 years. If anything is not supported, just go with the regular API. If anything breaks with future updates, just remove RockMigrations from the project and that's it. From that time on you could rely on core api and just remove old code (if that's even necessary).

But there's nothing bad in using the core api if you prefer. It's just too cumbersome for me.

[jploch]

@bernhard Thanks for the quick reply. I just watched another of your videos and together with your explanation I can see the benefits. It looks really nice! I just discovered that you now can copy the code directly from the templates and fields inside the GUI, which makes it even easier to use.

You also have a good point about RockMigrations not really breaking stuff as it's just used for the migration and can be replaced later on if necessary. I will test it a bit more and see if it makes sense for my next project.

[bernhard]

It seems to be a common misconception that if you start using RockMigration there is no way back and you need to use RockMigrations for everything. That's not the case.

[gebeer]

1 hour ago, jploch said:

I will test it a bit more and see if it makes sense for my next project.

RockMigrations makes total sense for all the projects I'm using it on 🙂

[bernhard]

19 hours ago, jploch said:

I just discovered that you now can copy the code directly from the templates and fields inside the GUI, which makes it even easier to use.

Recent versions of RockMigrations also show you all the settings and their possible values once the module is installed just by hovering over the setting. I'd recommend using this technique rather than copying all the code. Most of the time that is easier and you don't end up with messy migration code...