"Continuous integration" of Field and Template changes

[Rob]

I guess we have no further development on this.

I suspect that the best option for most people is to just try to get their repeated-work process as painless as possible.

One option I have used before is to actually have the entire workflow (including any work/edits that the "client" does) handled on a staging server/site and then just manually or automatically (cron job) mirror the entire thing to a live server/site.

If you do it the manually-triggered-mirror way, you can still develop in a relatively safe place and let the client make edits etc, and perhaps just give them a simple one-button mechanism to trigger the mirror script.

Unfortunately I think that anyone who has been around the block for a few years (myself included) would testify that there is no easy way to do this stuff and that it depends on the site/context etc.  I've certainly never seen a perfect system that didn't have (at least) occasional problems  or require some level of manual work for the developer.  Not when it comes to anything more complex than a simple blog anyway.

Throw e-commerce or large traffic into the mix and it gets even more painful!

Perhaps time would be better spent discussing our own workflows so at least people have some ideas they can draw from.  It seems a shame for any of the talented devs here to spend too much time banging their heads against a wall to only come up with part solutions or sidesteps that don't really solve the inherent problems of content management and deployment.

[pwired]

Photoshop has a way of recording every step you do in your editing that you can save as a script.

You can export this script for playback the exact same edit steps in another photoshop install.

Of course photoshop has nothing to do with processwire cms but what about a script that records

all steps you do in editing a local copy of your website on your laptop. Then export the script and

play it back on the live server. Would such a script work as part of the processwire core with

export - import - playback possibility ?

Perhaps time would be better spent discussing our own workflows so at least people have some ideas they can draw from.

I simply open 2 pages in firefox. One for the local website copy on my laptop and a second page for the live server. Everything I edit in the local firefox page I copy in the on-line firefox page.

[mindplay.dk]

I haven't given up on this, and I came quite far.

It's been a couple of months since I worked on it - last couple of months have been kind of hectic for me, so I haven't made any recent progress.

I do want to work on this, and still believe it's possible - since I gave up on using $trackChanges, the approach I'm using should be pretty safe.

Probably a couple of weeks of work left on this at least though, and not sure when I'll get back to it...

[mindplay.dk]

Okay, I did some work on this today - adding, updating, renaming, deleting and changing the type of Fields now records and repeats.

I have only done some superficial testing of this: created and deleted some fields, changed properties on various tabs, renamed a Field, deleted one, and was able to repeat all of those actions, without problems. Tables were correctly created, renamed and deleted during those operations.

I'm impressed with how robust ProcessWire is in this regard - because the important stuff really does happen at the API-level (rather than in controller/action methods) all the operations are easily repeatable and the results appear to be consistent

Nice work @ryan !

Now, there is one thing that concerns me.

        $this->addHookAfter('Fields::load', $this, 'hookFieldLoaded');
        $this->addHookAfter('ProcessField::fieldSaved', $this, 'hookFieldSaved');
        $this->addHookAfter('ProcessField::fieldAdded', $this, 'hookFieldAdded');
        $this->addHookAfter('ProcessField::fieldDeleted', $this, 'hookFieldDeleted');
        $this->addHookAfter('ProcessField::fieldChangedType', $this, 'hookFieldChangedType');

I'm hooking into Fields::load to obtain a copy of all the Field properties before any changes can be made to these.

But in order to capture the changes, I had to hook into the ProcessField::field* methods, which appear to be there for the purpose of monitoring these events.

My concern is that those events only fire when changes are made via the ProcessField controller - and that this is not (necessarily) the only way to make changes to Fields. (I no longer recall why I had to use the controller-level events, as opposed to hooking into the Fields collection-level events - I will investigate this further the next time I work on this.)

I would really like to hear from @ryan on this one before I continue - there are currently no equivalent controller-level events for Templates (e.g. in ProcessTemplate) as far as I can tell? So currently the only option is to hook directly into the Templates collection methods.

[ryan]

This sounds awesome mindplay.dk!

My concern is that those events only fire when changes are made via the ProcessField controller - and that this is not (necessarily) the only way to make changes to Fields.

That's a valid concern, since modules or other API usages that are adding/manipulating templates/fields would not be triggered from the Process ones. Though if you only wanted to capture changes made interactively, that would be the place to do it. 

Attached are replacements for /wire/core/SaveableItems.php and /wire/core/Fields.php. They contain additional hooks, which I think may be more what you are looking for. Let me know if these work out, and I'll commit them to the core. I'm a little short on time this afternoon, so haven't had a chance to try them myself, though the additions are very minor and simple so can't think of any reason why they wouldn't work. 

mindplay.zip

I added a few hooks to SaveableItems.php which is used by both Fields and Templates. You can hook these either by wire('fields')->addHook('methodToHook', $this, 'yourMethod') or $this->addHook('Fields::methodToHook', $this, 'yourMethod'). To operate on templates, you would replace 'fields' with 'templates'. Here are the hooks, and they work exactly like their Pages class counterparts:

public function ___saveReady(Saveable $item);
public function ___deleteReady(Saveable $item);
public function ___saved(Saveable $item);
public function ___added(Saveable $item);
public function ___deleted(Saveable $item);

These two were also added to the Fields class (in the attached zip):

public function ___changedType(Saveable $item, Fieldtype $fromType, Fieldtype $toType);
public function ___changeTypeReady(Saveable $item, Fieldtype $fromType, Fieldtype $toType);

For all of those above, it doesn't matter if you hook before or after. For the existing hooks that were already there, it does matter whether you hook before or after. Though not sure if you necessarily need any of these:

protected function ___changeFieldtype(Field $field); // Fields class only
public function ___saveFieldgroupContext(Field $field, Fieldgroup $fieldgroup); // Fields class only
public function ___clone(Saveable $item); 
public function ___delete(Saveable $item);
public function ___save(Saveable $item);
protected function ___load(WireArray $items, $selectors = null);  

Let me know if there is anything else I can add/adjust to facilitiate what you are working on.

[mindplay.dk]

Looks like added() and saved() are both being called on save() ?

It's a bit confusing with the high number of events - I can't imagine I'll need all those? I'm really not a fan of the word "ready", it's meaning is never really clear - ready for what? ... is it a good idea to have so many of these "vestigial" methods?

I think I'll just take a stab at this working from the current code in Git - and then see if I need any new events, how does that sound? I'd rather not bear the guilt of making you muck up your elegant tidy codebase with all this junk just for my sake

[ryan]

This makes the events consistent with how they are in the Pages class, which I think is a good thing. They are more reliable to standardize upon because, for example, saved() is called only after an item has been saved without error, and saveReady() is called only after its been confirmed the item will definitely be saved (no conditions to prevent). Without these, your hooked functions have to consider these things themselves (at least, if you want them to be thorough). Granted, there are more potential conditions within Pages than here, but these are hooks that probably belong here either way. I was just looking for a good reason to add them. While I think you may be able to do what you need without them, feel free to use them if helpful, because they belong here.

[mindplay.dk]

My problem is this:

class Collection
{
    public function save()
    {
        if (conditions) {
            onBefore()
            doStuff()
            onAfter()
        }
    }
}

Because the event system already uses the before and after concepts, it gets confusing - the above could be simplified as:

class Collection
{
    public function save()
    {
        if (conditions) {
            $this->doStuff()
        }
    }
}

Because I can already hook doStuff() both "before" and "after", you just need an inner and an outer method. "ready" is just a synonym for "before". You said yourself, it doesn't matter if you hook those "before" or "after", because nothing actually happens inside them...

[ryan]

I understand what you are saying. But the reasons are about consistency, readability, DRY and flexibility.

For consistency, we want to have similar/identical hook naming conventions across Pages, Templates and Fields, and was just looking for the right opportunity.

For readability, we want people to hook into things that that make sense purely by name. These hooks don't really cost us anything in terms of resources, but reduce potential ambiguity for the developer. I generally prefer more methods to more arguments, if that is an indicator. 

For DRY, I don't want everyone's save() hook to have to check return values or other things before deciding if it can proceed. I don't want people to have to devise ways of determine if a page was just added or already existed. Basically, I want to offer very convenient "sure things" in terms of hooks, where people don't have to think about internal conditions, and don't have to write code to accommodate them. 

Lastly, is flexibility for those of us that get deep in module development. I don't want 99% of people to have to think about internal conditions (per DRY above). But at the same time, I want to give the other 1% the flexibility to do so when their need calls for it. For instance a before(save) hook might be more appropriate than a saveReady() hook if I wanted to modify the conditions that will be present. Likewise, an after(save) hook might be more appropriate than a saved() hook if I wanted the ability to take action when something couldn't be saved. 

[mindplay.dk]

In Fields.php#415, the invokation of changedType() doesn't match the hook method - you forgot to pass in the $fromType and $toType arguments.

EDIT: I added the missing $fromType and $toType arguments and integrate the new hooks - I noticed one important change immediately, when changing the Field-type, there are actually two events during the same request, first changedType() and secondly saved(), and those get correctly recorded as one migration, in the correct order. So this works much better and is surely much safer than the controller-hooks. Thanks!

[ryan]

Not sure how I missed that. I will correct and update to the dev branch today. Thanks! 

[mindplay.dk]

FYI, this is still happening - made good progress on capturing and repeating changes to Templates today.

I expect this will take a while though, as this is not quick and simple by any means - this is still a spare time project, and I have very little time at the moment...

[apeisa]

Thanks for the update Rasmus. Eagerly waiting and definitely paying (if you think about making it commercial).

[mindplay.dk]

I'm thinking dual licensing at this point, but we'll see. For the moment, I don't have a working product yet

[mindplay.dk]

Ryan, I just pulled the latest dev-branch... Templates::saved() and Fields::saved() etc. are now firing again, but they're now firing twice every time I save. What gives?

Edit: I attempted to debug_backtrace() to see where the call is coming from, but that just results in "Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 123473921 bytes)" - something causes infinite recursion internally in debug_backtrace() so tracing this seems impossible...

[ryan]

I'm trying to reproduce this so far, but can't. I added this code to my /site/templates/admin.php, but only get 1 message when I save a template or field: 

$templates->addHook('saved', function($event) {
  $this->message('Templates "saved" hook executed');         
}); 

$fields->addHook('saved', function($event) {
  $this->message('Fields "saved" hook executed');         
}); 

Are there any other factors, like other modules or is it possible your hook is getting attached twice?

[mindplay.dk]

I haven't worked on the project since we last discussed it, and it worked fine up until I switched to the master release where those hooks were missing.

Note that I'm attaching my handlers statically:

    public function __construct()
    {
        $this->addHookAfter('Fields::load', $this, 'hookFieldLoaded');
        $this->addHookAfter('Fields::saved', $this, 'hookFieldSaved');
        $this->addHookAfter('Fields::added', $this, 'hookFieldAdded');

        $this->addHookAfter('Templates::load', $this, 'hookTemplateLoaded');
        $this->addHookAfter('Templates::saved', $this, 'hookTemplateSaved');
        $this->addHookAfter('Templates::added', $this, 'hookTemplateAdded');
    }

[ryan]

I ran my tests attaching them statically too, but it doesn't make a difference. I will try bundling them into a module (rather than anonymous functions from a template file) to see if it makes any difference. But two suggestions I have here are: 

1) attach your hooks in an init() method, rather than _construct(). ProcessWire may still be booting when your _construct() is called. Though in this case, I doubt it makes any difference, but I would try that just in case. init() or ready() is typically where hooks are attached (initializing connections with external things), whereas _construct() is for initializing internal things. init() is called before the current $page is known, and ready() is called after it is known and available as an API variable. In your case, init() is the right one to use (though ready() could also be used). 

2) after moving them to an init() method, switch to using non-static -- attach directly to $this->fields and $this->templates (the API variables). Static hook attachments are only necessary for objects that might have more than one instance, like Page, User, Field, Template, etc. But there will never be more than one instance of Fields or Templates, so you can benefit from slightly less overhead by attaching them directly to the API variables. 

[mindplay.dk]

@ryan init() is too late for the load() hook, as Fields at least (not sure about Templates) have already loaded at that time. That's why I'm forced to attach my hooks during __construct() rather than init().

Like any regular module, I had my hooks defined in init() initially, but it doesn't work - I just retested, and it still doesn't work. The load() hook never fires.

I tried changing to non-static hooks in init() as well - no difference.

I also tried non-static hooks in __construct() and that doesn't work - the hooks don't fire at all...

[ryan]

Sounds like it's a mystery. When you've got the code at a point where I can play with it, let me know and I'll try to debug it here. Or if you've got a skeleton of it that reproduces the issue, that's just as good. 

[mvolke]

1) Migrations

I had the same problem during my work with Processwire. If you have a team working on a Processwire project where everyone has his own dev server then you need to be able to version the "structure" you put into Processwire and keep it in sync with the template files. I found it to be important to strictly distinguish between structure and data. In 90% of the cases this means templates and field can be seen as structure and pages would be the data.

In my current project I tried to use the concept of migrations with Processwire utilizing the phpmig library.

If you strictly create all of your "structure" with migrations and never with the Backend UI then it works perfectly, your structure is always in sync with the template files and you can put both in your favorite VCS.

To give you an example of how this works:

https://gist.github.com/webholics/6191779

2) Structure vs. Data

I'd like to add more thoughts to the problem of mixing structure and data in CMSes. This is not a problem only Processwire has, but most CMSes do this (in my opinion) wrong. If your main target user group are developers, it should be possible to keep those things separate in order to enable professional workflows like continuous integration and TDD. This is only possible if you define the structure in code or in config files and not in the database and via a backend UI. It is always the case that template files and structure are strongly depended on each other. 

One CMS - I'd like to mention here for inspiration - which implements this concept perfectly is Bolt (http://bolt.cm/) .

They use YAML files to configure the database models. Bolt then tries to keep them in sync with the database.

Maybe it helps to have a look at how they did it.

[apeisa]

Django does it well too: syncing models from code to database. https://docs.djangoproject.com/en/dev/topics/db/models/

[mindplay.dk]

@mvolke the module I'm working on synchronizes structure only - Templates, Fields and Fieldgroups. It stores captured model operations in JSON files.

I currently have no plans to synchronize content (Pages) though it may prove to be necessary at some point - because Pages are used for so many things, certain Pages may need to synchronized, for example options for drop-downs.

I'm having Ryan take a look at the module now, and his initial reaction was positive - I think we can make this work, and I think it'll work well.

This is only possible if you define the structure in code or in config files and not in the database and via a backend UI.

You would think that, but if you look at the ProcessWire codebase, the entire meta-model, with all possible operations, is encapsulated and supports hooks - so it is actually perfectly feasible to implement this in ProcessWire.

As proof of concept, I already have all Field operations captured and repeatable. Because this is implemented at the lowest API level, it is actually independent of controllers and user-interface - that is, if you were to build your own admin modules that (for some reason) make changes to any part of the meta-model, those changes would be correctly captured and would be repeatable, independently of any admin UI.

There is still substantial work to do on this module, but I would say it's about half-done at this point, and there are no major roadblocks to completion - the fundamental idea is proven and works, so it's a matter of building it out completely.

[mvolke]

@mindplay.dk This sounds really great. I think it could be the perfect solution in the long run. Until this module is finished and tested, I can recommend the phpmig migrations approach.

[Hari KT]

Hi I was just wondering what you guys think about this

https://github.com/ryancramerdesign/ProcessWire/pull/273

And you can use a tool like dbv or something similar .. Thoughts ?