Migrating modules

[Aaron Ahearne]

Hi,

I've recently started on a team using Processwire, so Im quite new to it but I've had plenty of experience with Symfony2.

We have an agency licence for the FormBuilder module and we want to use some sort of CI to deploy.

When you first install FormBuilder module, it creates some new tables and requires some set up via a GUI including adding a licence key. We have looked into containing our licence key in a config file, then using some JSON files to sync to the database. This doesn't give us a lot of control and requires a fair bit of custom migrations.

• Is there a way to deploy a ProcessWire site including the FormBuilder module without going through the GUI setup every time?
• If not, Are there any migration modules for ProcessWire?
• If not, are there any recommended third party migration tools that are frequently used with ProcessWire?

Many thanks!

[adrian]

Module Toolkit does exactly this, and more!

https://processwire.com/talk/topic/8410-module-toolkit/

It's still not in a release worthy state - some functionality needs some tweaking - it's on my list, but you know how it goes

Still worth checking out though as I think it should handle your needs very well.

PS Welcome to PW and the forums!

[LostKobrakai]

If you really want to use automatic deployment you need to do it the same way as in every other framework and not use any GUI at all for everything which needs to be created in the deployment. I recently created just a small process module, that does list all migration classes in a certain folder, which do just have a update and downgrade function. With the api and some small helper functions one can mimic the things you do in the GUI quite easily and you've the benefit that every change is also source controlled (and/or versioned). I'm currently using it manually, but these can also be called automatically by a deployment script. 

[Aaron Ahearne]

Adrian that is almost what I was looking for but there is a lot of GUI interaction still. The code is still helpful and I may be able to use the export function with some custom code to automatically load in the modules.

LostKobrakai your module sounds exactly like what I need. Do you have a module or a github page where I could take a look at your process? Adding in full migrations would be ideal and we can definitely call them from a deployment script.

The only issue I can see with both solutions is that it migrates the config, so I assume it is storing the licence keys in a database migration file (as the licence keys are part of the module config object contained in the db). If this is the case, they will definitely need extracting to a file that is not synced with the repository.

[LostKobrakai]

The module is quite uninteresting. It's just generating a list of those mentioned classes. But I can show you what I mean by migration (not db migrations, which would be tedious as shit). The Migration class is just an abstract class, that does make sure everything is according to contract and holds some helper methods not used here. And it's extending Wire so all the processwire api variables are injected properly.

class Migration_2015_10_22_13_15 extends Migration {

	public static $commit = "";
	public static $description = "Install MapMarker\nCreate map fields";

	public function update() {
		$this->modules->install("FieldtypeMapMarker");

		$from = new Field();
		$from->type = $this->fieldtypes->get("FieldtypeMapMarker");
		$from->name = "travel_from";
		$from->label = "Start";
		$from->defaultLat = "…";
		$from->defaultLng = "–";
		$from->defaultType = "ROADMAP";
		$from->defaultZoom = 14;
		$from->save();

		$fg = $this->templates->get("invoice")->fieldgroup;
		$last = $fg->get("annotation_internal"); //annotation_internal
		$fg->insertAfter($from, $last);
		$fg->save();
	}

	public function downgrade() {

		// Remove fields
		$f = $this->fields->get("travel_from");
		foreach ($f->getFieldgroups() as $fg) {
			$fg->remove($f);
			$fg->save();
		}
		$this->fields->delete($f);

		// Remove module
		if($this->modules->isUninstallable("FieldtypeMapMarker")){
			$this->modules->uninstall("FieldtypeMapMarker");
		}
	}

}

Edit:

This is more about building up the same state everytime instead of migrating a current state over to another installation. 

Edited November 6, 2015 by LostKobrakai
Shortend the code a bit more

[kongondo]

Not exactly what you are looking for but just thought to mention the Wireshell module

https://processwire.com/talk/topic/9494-wireshell-an-extendable-processwire-command-line-interface/

Welcome to PW and the forums!

[ryan]

Aaron, PM me your FormBuilder key and I can reply with a snippet of code that will do it for you. 

[Aaron Ahearne]

Thanks LostKobrakai, thats really helpful. I can use adapt this into roughly what I need. That looks like a Symfony migration though, are you using this code from a separate system? You're extending Migration (thats why I assume Symfony) and using $this, which would have no scope outside of this code. Just curious.

 That CLI tool will definitely come in useful at some point, thanks kongondo.

Ryan, I'd love that code snippet, unfortunately its not my FormBuilder key so I dont want to risk giving it out, even though Im pretty sure you are the legit Ryan!

[LostKobrakai]

The extend is basically just an replacement for an interface, which conveniently can hold some helper methods I'm using. It's also extending Wire so all the ProcessWire api variables are injected and accessable by e.g. $this->pages or $this->templates. There's nothing framework behind this. The execution does in my case happen in an Process module, that does just list all the available migration classes and shows and update/downgrade link to run them. But you can call those classes anywhere, just make sure to bootstrap processwire, so the api is available, before executing any of the migration methods.

abstract class Migration extends Wire{

	public static $commit;
	public static $description;

	abstract public function update();

	abstract public function downgrade();

	[…] // Few non abstract helper methods
}

[Aaron Ahearne]

Thanks for your help guys. I've managed to get something going, just on one last section that Im struggling with.

A very simple version of what I have done:

  require_once("./modules/FormBuilder/FormBuilder.module");

  $formBuilder = new FormBuilder();

  $formBuilder->___install();

 

I have done it this way as I am storing the formbuilder files in the GIT repo, this way I just need to tell the server to install them the first time around.

 

I'm having some trouble getting the modules table to populate. I was previously setting it manually with raw SQL, but I want to use the API. My code is:

 

$modules->___saveModuleConfigData($class, $data);

 

The class is "FormBuilder" and the data is just a string. But I get the error:

 

Uncaught exception 'WireException' with message 'Unable to find ID for Module 'FormBuilder'' in /vagrant/oliver2/html/wire/core/Modules.php:2544

 

I have tried using $modules->___install("FormBuilder") and using $modules->get("FormBuilder") to get it to install the config in the modules table, but no luck. Trying to use $modules->___install() returns null because it stops at line 1255:

 

if(!$this->isInstallable($class)) return null;

 

I'm really stuck on what I need to do to get this module config installed. Any ideas?

 

[LostKobrakai]

Why not simply $this->modules->install("FormBuilder");? And to set the modules settings (api key and so on), see this: https://processwire.com/talk/topic/648-storing-module-config-using-api/

Also just to make this clear beforehand, hookable function (these with the three underscores) are called without those underscores. These are only there to mark those functions as hookable.

[Aaron Ahearne]

Thanks LostKobrakai thanks for the tips. I've been looking at that forum post for the whole day trying to figure out why mine isnt working.

FormBuilder isnt actually showing up in my modules table at all, Im wondering if this is why it cant find it. Even using wire("modules")->saveModuleConfigData("FormBuilder", $data); it is telling me that it cannot find an ID for FormBuilder.

Im guessing this row should be created via ___ those hooks when FormBuilder installs itself?

 

FYI, Im using all of these functions outside of any scope pretty much, just a plain PHP script. Im not sure its a possibility to use it inside its own class, but wire() is resolving to an object, so Im guessing it is somewhat inside of a ProcessWire scope.

[LostKobrakai]

Could you just post that part of your code, that will make things easier and more clear to explain.

[Aaron Ahearne]

echo "Starting...\n";

include_once("../index.php");

if (!$modules->isInstalled("FormBuilder")) {

require_once("./modules/FormBuilder/FormBuilder.module");

include_once("./install-config.php");

echo "Installing Form Builder\n";

echo "-------------------\n";

if (!file_exists("./install-config.php")) {

echo "install-config.php not found";

die();

}

// Get license key from config file

if (!$config->formBuilderLicenseKey) {

echo "Missing Form Builder license key";

die();

}

// Create cache directory

$filesPath = "./assets/cache/form-builder";

if (!is_dir($filesPath)) {

if (!is_dir($filesPath)) {

if (!is_dir("./assets/cache")) {

mkdir("./assets/cache");

}

mkdir($filesPath);

echo "Creating formbuilder cache directory at " . $filesPath . "\n";

}

}

$formBuilder = new FormBuilder();

$formBuilder->___install();

$data = json_decode(file_get_contents("./metadata/formBuilder/modules.json"),

true);

/**

* Install config data

*/

if ($data) {

foreach ($data as $class => $data) {

if ($class === "FormBuilder") {

$data['licenseKey'] = $config->formBuilderLicenseKey;

}

wire("modules")->saveModuleConfigData($class, $data);

}

closedir($handle);

}

}

Edited November 10, 2015 by LostKobrakai
added code block

[LostKobrakai]

I've directly commented my changes in the code.

<?php
echo "Starting...\n";
include_once("../index.php");

/**
 * API Variables are only available through the $wire variable 
 * or the wire() function. https://processwire.com/api/include/
 */
if (!wire('modules')->isInstalled("FormBuilder")) {
  /**
   * Not needed
   */
  // require_once("./modules/FormBuilder/FormBuilder.module");
  include_once("./install-config.php");

  echo "Installing Form Builder\n";
  echo "-------------------\n";

  if (!file_exists("./install-config.php"))
    die("install-config.php not found");

  /**
   * I hope $config is a variable comming from install-config.php
   */
  if (!$config->formBuilderLicenseKey)
    die("Missing Form Builder license key");

  /**
   * No need to do this manually, let FormBuilder install itself
   * 
   * Hint: PW does have some neat helper functions for filesystem task if you need them. 
   * See /wire/core/Functions.php for them.
   */
  // Create cache directory
  // $filesPath = "./assets/cache/form-builder";

  // if (!is_dir($filesPath)) {
  //   if (!is_dir($filesPath)) {
  //     if (!is_dir("./assets/cache")) {
  //       mkdir("./assets/cache");
  //     }
  //     mkdir($filesPath);
  //     echo "Creating formbuilder cache directory at " . $filesPath . "\n";
  //   }
  // }

  /**
   * Refresh modules and install FormBuilder
   */
  wire('modules')->resetCache();
  $formbuilder = wire('modules')->install("FormBuilder");

  if(is_null($formbuilder))
    die("FormBuilder wasn't installed correctly");

  $data = json_decode(file_get_contents("./metadata/formBuilder/modules.json"),
    true);

  /**
   * Install config data
   */
  if ($data) {
    foreach ($data as $class => $data) {
      if ($class === "FormBuilder") {
        $data['licenseKey'] = $config->formBuilderLicenseKey;
      }
      wire("modules")->saveModuleConfigData($class, $data);
    }
    /**
     * Not needed and what's $handle?
     */
    //closedir($handle);
  }
}

[Aaron Ahearne]

Thank you so much for taking the time to look at this. Your comments are really helpful.

Unfortunately, at the moment the following line is still returning null.

$formbuilder = wire('modules')->install("FormBuilder");

Im going to have a further look into it and take a further look into the docs in your comments and see if I can figure out what is going on.

[LostKobrakai]

That's strange as the admin interface probably does use that exact method when installing a module. Can you install FormBuilder via the gui in that stage?

[Aaron Ahearne]

Yes, I am able to install via the admin interface. Im wondering if I've just corrupted the install from my previous attempt.

I've found the problem. In Modules.php line 1221.

$installable = array_key_exists($class, $this->installable); 

I can see that $this->installable essentially has a list of installable modules. FormBuilder does not appear in this list. Im going to do a fresh install of ProcessWire now to see if that fixes my problem.

[Aaron Ahearne]

Ok so that has fixed it.

As suspected, the previous install attempts had caused $this->installable to be populated incorrectly. Although FormBuilder did not appear in the Modules table, it was still being detected as installed. I'm not sure how the installable array in Modules is being populated so I don't know the root cause of it.

Thanks so much for your help on this. I'd been through all of the tutorials but this has given me a far greater understanding of PW. I was even able to remove some of the data from my stored config as this final approach auto populated some of it for me.

[LostKobrakai]

Yeah, it's essentially doing the same as in the GUI, so it does have all the default settings already. If you want to understand the api of processwire it can be quite enlightening to just look into the core modules and see how things are done there.

[LostKobrakai]

I've released the module mentioned above as beta: https://processwire.com/talk/topic/13045-migrations/