Development/Staging of Processwire site

[Bjorn]

Before getting to my question: Sorry if this has been discussed already...

I am reasonably new to using Processwire, but think it's absolutely great! I have built my site using it and will be moving it to a production environment for use by various people. 

My question is how to go about the ongoing development of the site when I use a development/staging area and not loose data in the production environment when I have to update?

As part of my ongoing development, I will be editing template files, possibly adding modules, configuring modules and making changes in the database i.e. new fields, modifying fields etc. When I need to update the live server, inserting/updating the files on the filesystem is easy enough (will probably use Git) - but how to update the database when users have been making their own updates & content via the front-end? Obviously my dev-version of the database will not have this new content... Re-playing all the modifications on the production server (i.e. config of modules, updating fields etc.) is also not really an option since this is very error-prone and a tedious affair.

Can someone possibly help me here?

[diogo]

The system that's proven more effective for me is:

1 — do all the non destructive database changes on the live site (adding fields and adding them to templates, creating new templates and so on)

2 — import the database to the development server and do the templates changes

3 — pull the new files to the live site without touching the database

4 — do some cleaning on the live database if needed (remove fields, templates and so on)

Another tip, although this one doesn't really answer your question, just because I think it can be useful

One system that I already used effectively was to create another install of PW (a copy of the other) in the live server (on a subdirectory or subdomain) that connects to the same database, and develop all the changes on the template files right there, using real and up-to-date content. Then I simply replace the template folder by the new one.

[LostKobrakai]

I'm using simple migration scripts, where I update anything via the api. It's a bit verbose and not as nice as using the backend, but it keeps db changes in a manageable and documented fashion. Also after a while it's more copy paste for most tasks.

[elabx]

I have the vague impression that database migrations are better supported in other types of frameworks/languages (Rails/Ruby)? Or is Processwire simply not intended to have this type of implementation? Or am I misinterpreting something here? I can only guess from my experience, that database sync is not trivial.

It just sparks my curiosity that currently there are two threads asking about this, and also have read about a lot of people rather using static CMS due the "databaseless" way these apps work.

[diogo]

I have the vague impression that database migrations are better supported in other types of frameworks/languages (Rails/Ruby)? Or is Processwire simply not intended to have this type of implementation? Or am I misinterpreting something here? I can only guess from my experience, that database sync is not trivial.

It just sparks my curiosity that currently there are two threads asking about this, and also have read about a lot of people rather using static CMS due the "databaseless" way these apps work.

It's an old problem, and no, there isn't a simple solution. I think the most commonly adopted solution is still this one, back from 2011 http://ben.kulbertis.org/2011/10/synchronizing-a-mysql-database-with-git-and-git-hooks/

[LostKobrakai]

Database migrations are "better" supported in other frameworks because they wouldn't function without them. A laravel framework doesn't have any databases beside those setup by migrations. But these are actually really bare bone in that they mostly update schema changes. For migrating db rows most use database seeding libraries, which then create the necessary models to be stored. Both can be done with the processwire api without actually needing to learn something new. 

This is basically what model factories do in frameworks (besides the autofill functionality). 

$pages->add('basic-page', $pages->get("/url/"), 'profile', array(
  "someField" => "value",
  "anotherField" => 202
);

And this may be what migrations do.

// Create field
$f = new Field();
$f->name = 'text';
$f->type = 'FieldtypeTextarea';
…

$f->save();

// Add to template
$bpt = $templates->get('basic-page');
$bpt->fields->add($f);
$bpt->fields->save();

[Bjorn]

Thanks for the comments guys! What diogo wrote is pretty much what I am doing now - but that would imply that I lock down the live site whilst I am developing, which is not ideal.

I do like LostKobrakai's proposal which I never considered before. It would make it slightly lengthy to build the instructions and more so to test it (when it fails, you have to restore DB backups, correct the instructions and then try again).

 

Indeed it looks as if there is no silver bullet here. Pity.

[diogo]

The idea of the process that I described is exactly to avoid locking the live site. That's why doing all the nondestructive changes on the live site and avoiding any DB changes on the local copy.

[pwired]

I'm using simple migration scripts, where I update anything via the api. It's a bit verbose and not as nice as using the backend, but it keeps db changes in a manageable and documented fashion. Also after a while it's more copy paste for most tasks

Can you show some examples of this ?

This would be handy to update only parts of the online database and leave the rest alone.

Let's say you develop local on your laptop and want to update the database of the live site on the host.

If you have a guestbook, pagecounter, subscriptions, etc on the live site then these will be overwritten

in the online database each time you update from your local dev site.

[SteveB]

Before PW my approach was to keep data of the content and data about the system in separate tables (and files). That helped a lot.

Given PW's great API, migration scripts is the way to go. I'm not there yet but I've been doing massive data imports using scripts and API and am very pleased with how well that works.

The challenge is keeping track of what you did while trying out different approaches and refining things. Probably makes sense sometimes to have Sandbox, Staging, Live. Do the messy creative work in the sandbox. Reproduce those changes deliberately and systematically on a fresh copy (Staging) using your scripts. Repeat as needed. Then (make a backup and) apply scripts to Live.

[LostKobrakai]

@pwired

There's nothing fancy about it. I've a small process module as well, which does the execution of the functions as well as storing which migrations are migrated and which aren't. 

// Migration.php
abstract class Migration extends Wire{

	public static $description;

	abstract public function update();

	abstract public function downgrade();

}

// 2016-03-03_18-27.php

class Migration_2016_03_03_18_27 extends Migration {

	public static $description = "Update Homepage title";

	public function update() {
                $this->pages->get('/')->setAndSave('title', 'Super fancy new Homepage title');
	}

	public function downgrade() {
                $this->pages->get('/')->setAndSave('title', 'Home');
	}

}

[bernhard]

Thank you for the snippet lostkobrakai. Do you think it would somehow be possible or make sense to have kind of an export button (like the one at templates) that makes it possible to copy/download a script with all the API commands to create an exact copy of that field with all settings?

[pwired]

@LostKobrakai, thanks for coming back on this.

I guess if one understands the api, mysql and php enough, you could select only specific database tables to update,

leave the rest of the database tables alone and run your own database updater.

[LostKobrakai]

I've released the module mentioned above as beta: https://processwire....045-migrations/