RockMigrations1 - Easy migrations from dev/staging to live server

[bernhard]

1 hour ago, MarkE said:

However, being forgetful, I will need to document something for myself, so happy to share if it is useful.

PRs to the readme always welcome!

1 hour ago, MarkE said:

but I think attention should also be paid to the back-end and in particular to the issue of maintainability which your module addresses.

Agreed. I've mentioned that I'd vote for an API first approach. Build UI for beginners, but building it upon a solid API (something similar to RockMigrations) so that experts can just as easily (or even easier) use that tools for automation or advanced needs. PW goes that route very often. I hope that one day migrations adopt that approach as well.

[MarkE]

One (rather trivial) issue: if you attempt to migrate the home page, then PW throws an error because the parent is invalid. It seems to be necessary to unset the parent and not include 'parent' in the selector in RockMigrations::createPage().

[bernhard]

Thx for finding that. Could you please share your code that you are using so that I can easily reproduce it?

[MarkE]

1 hour ago, bernhard said:

Thx for finding that.

Actually it found me ?

My code could probably be improved, but here is wahat I did (from around line 1190)

 // if the page is the home page then we need to avoid referring to its parent when saving
      if ($parent === 0) {
        $parent = '';
        unset($data['parent']);
      } else {
        // make sure parent is a page and not a selector
        $parent = $this->pages->get((string)$parent);
      }
      // get page if it exists
      if ($parent and $parent->id) {
        $selector = [
          'name' => $name,
          'template' => $template,
          'parent' => $parent,
        ];
      } else {
        $selector = [
          'name' => $name,
          'template' => $template
        ];
      }

      $page = $this->pages->get($selector);

Hope that helps.

[bernhard]

I'm talking about the code of the migration that throws the error - not the code of your fix ? 

[MarkE]

1 hour ago, bernhard said:

code of the migration that throws the error

in json format - before conversion into array and used in migrate(['pages' => ...]) :

 "home": {
        "template": "home",
        "parent": 0,
        "status": 9,
        "title": "System",
        "summary": "This is the \"home\" page for the admin site - i.e. all other pages are children of it. It therefore holds some system-wide settings. You can change the title to suit any system-wide branding.\r\nSome of the fields (operator, websites and financial year) are duplicated on Property pages. The currently active property is shown in the top menu - if there is one and it has these duplicated fields filled on its page then they will take precedence, otherwise the values on this page will be used. The values on this page will be used if no property is active.\r\nYou can also manage image pages and management roles from this page.\r\nOne of the image pages is called \"letterhead\". Any images on that page will be used in letters/emails if no property is active or if the active property has no images on its letterhead page. You may create other image pages to hold images that you might want to use system-wide.",
        "operator": "/contacts/xyz/",
        "webmaster": "/contacts/xyz/",
        "website": "",
        "website2": "",
        "image_single": {},
        "taxYear": null,
        "dayOfMonth": "",
        "month": {},
        "taxYear_END": null
    },

but any home page would throw the error, I think.

[bernhard]

Could you please provide a reproducable code snippet that I can use on a fresh PW installation having RockMigrations installed?

$rm = $modules->get('RockMigrations');
$rm-> ... your code here ...

And maybe add what you are trying to achieve.

[MarkE]

10 hours ago, bernhard said:

a reproducable code snippet

$this->rm()->migrate([
            'pages' => [
                'home' => [
                    'title' => 'system',
                    'parent' => 0,
                    'template' => 'home'
                ]
            ]
        ]);

This just changes the title, for simplicity. The original changed other fields and was part of a bigger migration. But running just this should highlight the issue.

[bernhard]

Thx, that helped. The migrate() method was intended to create pages that do not exist in the system (eg pages for a blog).

I've made the setPageData method public, so if you want to set page data of the root page, use setPageData instead of migrate():

v0.0.34: https://github.com/BernhardBaumrock/RockMigrations/commit/801ac7ff94edbb7b4e430791874746580d995111

[bernhard]

v0.0.35 adds the file-on-demand feature. RockMigrations is now an autoload module, which makes it available as $wire->rockmigrations and makes it possible to attach hooks for several tasks. For example we could add hooks for GIT webhooks that trigger migrations after a push to a git repository...

Load files on demand from remote to local development

This idea comes from a blog-post of Ryan when he introduced the multiple hooks feature in 3.0.137: https://processwire.com/blog/posts/pw-3.0.137/#on-demand-mirroring-of-remote-web-server-files-to-your-dev-environment

This feature is now part of RockMigrations and setup is as easy as setting a config variable:

$config->filesOnDemand = "https://www.example.com";

[MarkE]

Hi @bernhard, just a simple question: Why in createField() - line 466 of RockMigrations.module, do you have

if(strtolower($name) !== $name) throw new WireException("Fieldname must be lowercase!");

I thought the rules were: upper and lowercase ASCII letters, digits and underscore.

Would it not be better to use $sanitizer->fieldName() or am I missing something?

[bernhard]

Hi @MarkE

That question ist not that simple ? I've had problems with non-lowercase fieldnames from time to time. I did not have time to dig into that in detail, so the quick solution was to force rockmigrations to use lowercase fieldnames. I know that RockFinder had problems with non-lowercase fieldnames, because the DB table name is always lowercase whereas the field name can also contain uppercase letters. But that problem was solved recently: 

So if you could provide simple and reproducable test cases and maybe a PR I'm happy to remove that restriction and allow non-lowercase fieldnames ? 

[MarkE]

9 minutes ago, bernhard said:

So if you could provide simple and reproducable test cases

Easier said than done. I'll see what I can do. I can see that if you are using RockMigrations from the start, then the issue does not arise as all fieldnames will be lowercase.

My situation is a bit different as I only use PW occasionally, and find the Admin UI, rather than a 'headless' approach, works better for me. However, I find migrations a pain. Consequently, I have started working on a UI interface module for your module, which is coming along quite nicely and which I will provide more details on once I have it in a reasonable state. It generates json files from the database which are then used as input to the migration, so it uses the existing fieldnames, which may not be lower case. Having run into this, as well as a few other problems with using RockMigrations in this way, I have switched to using the core code directly as I appreciate that RockMigrations was not designed with this in mind. For fields, I am using a modified copy of ProcessFieldExportImport::processImport() (although that has problems with options fields), for templates, I call ProcessTemplatesExportImport::setImportData() and am just using the standard API for pages. However, there are attractions in using the RockMigrations methods so I may return to this once all the UI stuff is working, particularly if you think it is a good idea.

[bernhard]

That sounds interesting ? Though I'm not sure if that approach can actually work well ? I mean... It can work for somebody of course, but I'm not sure how big the difference to existing solutions (field/pages import/export) would be? Maybe you could clarify that a bit?

But I don't want to discourage you from trying to find a way that makes migrations more easy to use. I know that this is a big hurdle at the beginning - it took me years to understand why Benjamin built his migrations module ? On the other hand migrations are a totally different concept compared to building websites via the PW backend... you need to take care of logical restrictions that sometimes arise (for example field settings depending on the existance of other fields or pages etc). That's why migrations SHOULD be code and that's why it is so hard to build some kind of UI or recording/diff tool for it...

Whenever I don't know the code for a migration I simply create the field and have a look at Tracys field code panel:

Then I copy the parts that I need to my migration and that's it. My IDE then tells me about the changes, I can commit them and everybody is happy ? 

But I'm really happy to see different approaches coming up and I'm looking forward to seeing any drafts ? 

[MarkE]

24 minutes ago, bernhard said:

not sure how big the difference to existing solutions (field/pages import/export) would be? Maybe you could clarify that a bit?

Well, no different, in a way, as it uses all that code. It just provides a UI to pull it all together and automate all the export/import in the right order. More later...

[dotnetic]

I try to setup a repeater matrix field. Can anybody like @aComAdi provide a working example, how to do this?

EDIT: It turned out, that I needed version 5 of the RepeaterMatrix field.

[dotnetic]

@bernhard Is there a way to generate and insert content into a repeater matrix item via RockMigrations?

First I generate the fields needed for the repeater matrix and add the repeater matrix field to a page. Then I want to create a new hero item on that page and insert some contents in it.

[bernhard]

I'm quite sure it is possible, but I don't have an example. I'm not using RepeaterMatrix any more and on my old projects I did not use it in combination with RockMigrations, sorry ? 

[bernhard]

The module gets better and better and I love working with it every day ?

v0.0.42 has a nice little update to include scripts or styles in the PW backend easily. Why is that great? Because you can organize your files better and have everything in place while still having the ease of use (and you don't need to install any other modules for such simple tasks):

Usage:

I'm using custom page classes for EVERY page I have in my PW installs. One might think that this is overkill - but I'm wasting so much time on old projects looking for code snippets that are all over the project. Some in ready.php, some in init.php, another story are JavaScript snippets for small GUI tweaks...

Not any more ?

I have a custom PageClass "Foo" and there I have a method that modifies the page edit form (ProcessPageEdit::buildForm) - this is great for changing field properties like columnWidth or labels etc. on the fly without ever having to fire a migration. If I want to change something for my Foo pages, I head over to my IDE, type "Foo" and get directly to the Foo PageClass.

But what about JavaScript? Today I needed a little snippet that populates some fields based on other fields. As easy as that:

// in the buildForm hook
$rm->addScript(__DIR__."/Foo.js");

Everything is well organized, because I have all my classes in a custom folder (inside a module, but the same would apply for the /site/classes folder):

.../myModule/classes
               |-- Bar.php
               |-- Foo.js
               |-- Foo.php
               '-- Whatsoever.php

And the JS file will only be loaded on the page edit screen to prevent overhead.

Why not just use $config->scripts->add() you might ask? Good question ? 

$rm->addScript() will additionally add a cache busting timestamp to the file and it will only include the file if it exists on the system. It will also work on Windows systems, which you might forget to take into account if quickly adding it via $config->scripts->add() and last but not least it will also work on PW instances that are running in a subfolder ? 

    if(!is_file($path)) return;
    $path = Paths::normalizeSeparators($path);
    $config = $this->wire->config;
    $url = str_replace($config->paths->root, $config->urls->root, $path);
    $m = $timestamp ? "?m=".filemtime($path) : '';
    $this->wire->config->scripts->add($url.$m);

Happy migrating ? 

[MarkE]

35 minutes ago, bernhard said:

I'm using custom page classes for EVERY page I have in my PW installs. One might think that this is overkill - but I'm wasting so much time on old projects looking for code snippets that are all over the project. Some in ready.php, some in init.php

I quite agree - I use custom page classes for practically every template. One question, though not entirely specific to RM: how do you deal with functions in init.php that are used by multiple classes? In my dbMigrate module in the 'parallel universe' of UI development ? I use some custom methods and functions that are in my DefaultPage.php and init.php because I use them more widely than just in the module. Of course, they can be readily found with the IDE (although it is an issue with packaging the module), but your comment about init.php made me think that I am missing a trick.

[bernhard]

26 minutes ago, MarkE said:

how do you deal with functions in init.php that are used by multiple classes?

I think I don't understand your question.. I don't have any functions in init.php nowadays. I have everything in init() of a module or in init() of a pageclass. If there is something in init() of a pageclass I have to trigger that manually, which is the reason for the loadClasses method of RM which triggers init() automatically https://github.com/BernhardBaumrock/RockMigrations/blob/2d85e460ce5aa394480906ad5b41cd1a0e86d0fe/RockMigrations.module.php#L1991-L2003

This is a simple pageClass I'm using in my CRM for managing contacts:

<?php namespace RockCRM;

use ProcessWire\HookEvent;
use ProcessWire\Page;

class Contact extends Page {

  const tags = RockCRM::tags;
  const tpl = RockCRM::prefix."contact";
  const prefix = RockCRM::prefix."contact_";

  const field_skills = self::prefix."skills";

  public function __construct() {
    parent::__construct();
    $this->parent = $this->wire->rockcrm->contacts();
    $this->template = $this->wire->templates->get(self::tpl);
  }

  public function init() {
    $tpl = "template=".self::tpl;
    $this->wire->addHookAfter("ProcessPageEdit::buildForm", $this, "buildForm");
    $this->wire->addHookAfter("Pages::saveReady($tpl,id=0)", $this, "onCreate");
  }

  /**
   * Page edit screen
   */
  public function buildForm(HookEvent $event) {
    $form = $event->return;
    $page = $event->process->getPage();
    if(!$page instanceof self) return;
    $crm = $this->wire->rockcrm; /** @var RockCRM $crm */
    $rm = $crm->rm();

    // modify page edit screen of a contact
    if($f = $form->get('title')) {
      $f->label = 'Name of the Contact';
      $f->notes = 'Enter a great Name';
    }
    
    // or load scripts with the new update
    $rm->addScript(__DIR__."/Contact.js");
  }

  /**
   * Migrate this pageclass
   */
  public function migrate() {
    $crm = $this->wire->rockcrm; /** @var RockCRM $crm */
    $rm = $crm->rm();
    $rm->migrate([
      'fields' => [
        self::field_skills => [
          'type' => 'text',
          'tags' => self::tags,
        ],
      ],
      'templates' => [
        self::tpl => [
          'tags' => self::tags,
          'icon' => 'user-o',
          'noSettings' => 1,
          'pageClass' => '\RockCRM\Contact',
          'fields' => [
            'title',
            self::field_skills,
          ],
        ],
      ],
    ]);
  }

  /**
   * Things to do when a contact is created
   * @return void
   */
  public function onCreate(HookEvent $event) {
    $page = $event->arguments(0);
    $page->status = 1; // published, non-temp
    $event->pages->names()->uniqueRandomPageName(); // unique name
  }

}

This setup is extremely powerful, extremely clear and extremely nice to work with ? 

Was that what you were asking for? ?

[MarkE]

20 minutes ago, bernhard said:

I think I don't understand your question.. I don't have any functions in init.php nowadays. I have everything in init() of a module or in init() of a pageclass. If there is something in init() of a pageclass I have to trigger that manually, which is the reason for the loadClasses method of RM which triggers init()

I guess that answers the question, thanks!

[bernhard]

Great update - working with access control got a lot easier today ??

Access Control

ProcessWire has a powerful access control system. When using RM to create new templates and pages it is quite likely that you also want to create roles and define access for those roles on the new templates. The basics of access control can easily be done via RM - for more advanced topics you might need to implement custom solutions. PRs welcome ?

Let's say we created an Events calender and we wanted to store all events (tpl event) under one page in the page tree. This page (tpl events) would only allow pages of type event. To manage those events we create a new role on the system called events-manager.

$rm->migrate([
  'templates' => [
    'events' => [...],
    'event' =>  [...],
  ],
  'roles' => [
    'events-manager' => [
      'permissions' => ['page-view', 'page-edit'],
      'access' => [
        'events' => ['view', 'edit', 'add'],
        'event' => ['view', 'edit', 'create'],
      ],
    ],
  ],
]);

On more complex setups you can use the API functions that are used under the hood directly. For example sometimes I'm encapsulating parts of the migrations into separate methods to split complexity and keep things that belong together together:

/**
 * Migrate all data-pages
 * @return void
 */
public function migrateDatapages() {
  $rm = $this->rm();

  // migrate products (having two custom page classes)
  // these have their own migrations inside their classes' migrate() method
  // where we create fields and the template that the class uses
  $product = new Product();
  $product->migrate();
  $products = new Products();
  $products->migrate();
  $rm->setParentChild(Products::tpl, Product::tpl);
  $rm->setTemplateAccess(Products::tpl, self::role, ["view", "edit", "add"]);
  $rm->setTemplateAccess(Product::tpl, self::role, ["view", "edit", "create"]);

  // same goes for all other data pages
  // ...
}

[bernhard]

v0.0.53
It always annoys me not to have the correct page name replacements for german umlauts...

Spoiler

Set german pagename replacements

$rm->setModuleConfig("InputfieldPageName", [
  'replacements' => [
    "æ"=>"ae",
    "å"=>"a",
    "ä"=>"ae",
    "ã"=>"a",
    "ß"=>"ss",
    "ö"=>"oe",
    "ü"=>"ue",
    "đ"=>"dj",
    "ж"=>"zh",
    "х"=>"kh",
    "ц"=>"tc",
    "ч"=>"ch",
    "ш"=>"sh",
    "щ"=>"shch",
    "ю"=>"iu",
    "я"=>"ia",
    ":"=>"-",
    ","=>"-",
    "à"=>"a",
    "á"=>"a",
    "â"=>"a",
    "è"=>"e",
    "é"=>"e",
    "ë"=>"e",
    "ê"=>"e",
    "ě"=>"e",
    "ì"=>"i",
    "í"=>"i",
    "ï"=>"i",
    "î"=>"i",
    "ı"=>"i",
    "İ"=>"i",
    "ğ"=>"g",
    "õ"=>"o",
    "ò"=>"o",
    "ó"=>"o",
    "ô"=>"o",
    "ø"=>"o",
    "ù"=>"u",
    "ú"=>"u",
    "û"=>"u",
    "ů"=>"u",
    "ñ"=>"n",
    "ç"=>"c",
    "č"=>"c",
    "ć"=>"c",
    "Ç"=>"c",
    "ď"=>"d",
    "ĺ"=>"l",
    "ľ"=>"l",
    "ń"=>"n",
    "ň"=>"n",
    "ŕ"=>"r",
    "ř"=>"r",
    "š"=>"s",
    "ş"=>"s",
    "Ş"=>"s",
    "ť"=>"t",
    "ý"=>"y",
    "ž"=>"z",
    "а"=>"a",
    "б"=>"b",
    "в"=>"v",
    "г"=>"g",
    "д"=>"d",
    "е"=>"e",
    "ё"=>"e",
    "з"=>"z",
    "и"=>"i",
    "й"=>"i",
    "к"=>"k",
    "л"=>"l",
    "м"=>"m",
    "н"=>"n",
    "о"=>"o",
    "п"=>"p",
    "р"=>"r",
    "с"=>"s",
    "т"=>"t",
    "у"=>"u",
    "ф"=>"f",
    "ы"=>"y",
    "э"=>"e",
    "ę"=>"e",
    "ą"=>"a",
    "ś"=>"s",
    "ł"=>"l",
    "ż"=>"z",
    "ź"=>"z",
  ],
]);

 

[bernhard]

v0.0.56

Today I had some troubles migrating field labels. Changes simply did not show up on the related page edit screen... turned out that some field setting overrides have been activated! Removing them via API is not that simple, so I added the feature to RM:

Quote

$rm->removeContext('yourtemplate', 'yourfield');

?