"Continuous integration" of Field and Template changes

[muzzer]

If you can get this module going it would be pretty sweet and I'd be happy to pay for this functionality.

In my experience with client-editable sites and local development the process usually follows a pretty regular pattern; Most updates are relatively minor, eg they may take perhaps 1 hour to 1 day max. In these cases its a matter of downloading the database from live (phpmyadmin), import into dev database (again phpmyadmin), do the alterations locally and then reverse the previous import/export back to live. Tedious but manageable

To make this process easier two modules would be adequate in my mind:

1. an easy export/download of the PW database from live (one-click from within the admin, not from phpmyadmin) and an easy import (again from within admin) of the download PW tables into the dev database.
2. A simple method to temporarily block all clients ability to login to the live site while development is underway, to prevent overwrites. If clients do try to login then a message "Sorry, you cannot login at this time as development is underway.You will be notified by email when this is complete. The estimated completion of development from now is 3 hrs and 14 minutes".
   Any client who tries to login will be added to a list and those clients automatically emailed when development is complete and the login block removed. To enable the block the admin (perhaps this could work in conjunction with the download module in 1 above) would simply enter a estimated time required for development  and hit "Block clients" button. The estimated time would be used to advise clients who try to login of how long they will likely need to wait before they can login

Just my two cents. I had a similar setup in the past in modx sites and it worked well for me.

[SiNNuT]

> I don't think anyone has proposed or suggested that? We're proposing a supplement/alternative, not a replacement. At least I would never suggest that, and I don't think that's what rajo was trying to imply.

I think that it was in reply of mvolke's post, which suggested using (.yml) files for defining content structure as the best option.

Apart from that, this is a an intetesting discussion. The idea of some sort of centralized facility to keep track of template and field changes, opening up a lot of options for dev work, seems quite nice.

I have yet to look into the other tools posted here, which take a different approach.

[mindplay.dk]

@muzzer you're describing the typical approach and that's what I'm unhappy with - deploying applications used to be like this, too...

While building mostly applications these past six years or so, continuous integration has become standard for me, more or less, and many clients expect it by now.

So it feels like a huge setback when you have to go and tell clients "we'll be down for half a day while upgrading your site".

Not to mention the extra half a day of work you need to bill for, effectively just to deploy.

And of course you can't work in teams anymore - one guy has to do the deployment work, and preferably the guy who did the implementation work, since he has the best odds of doing the work again without screwing up. So one person becomes a resource bottleneck, which isn't good in a company that serves many clients and needs to move fast.

Anyway, we all know about these reasons and others, I'm sure - the question is what can we do about it?

Once I've seen a better way, I'm just not the type of guy who is able to lean back and go "well, this sucks, but let's just keep doing it the old way" ;-)

By the way, the module basically works, but I just hate how much code it took, and the fact that it won't work for certain fieldtypes, and there's nothing I can do about it short of explicitly handling each fieldtype, possibly providing an API for thirdparty fieldtypes to integrate with the module etc. - it feels too brittle and halfbaked, and I'm not inclined to doing the rest of the grunt work to make the module usable, knowing that the idea is fundamentally flawed. I'm sort of manic that way ;-)

[Craig]

I think this is a very good discussion to have, and it's interesting to see how other people handle this situation.

As a developer, I'm often wearing one of two hats - one as a lead developer in the company where I work; and the other as someone who builds and maintains websites (using mainly ProcessWire) outside of work for organisations and people with a personal connection (or even just sites for myself).

At my company, most of our time is spent on software as a service apps built using CodeIgniter. We work locally on databases that match the structure of live ones. When we make schema changes in development, our tools (Adminer, HeidiSQL, phpMyAdmin etc) provide us with the SQL query that was executed. We put these in .sql "patch" files in the project repository (patch1.sql, patch2.sql...). We also have a custom script in the application which runs these patch files on-demand at the staging and production servers. The application database has a patch history table to keep track of what it has, and the script will then run the patch files it needs for the database schema to be "up to date".

As a team of three developers, this works well for us and the way in which we manage changes to those projects. The key to this process working for us is having the queries available and recording them appropriately when we make changes. They are plain text and tracked in source control.

I think what I described above is similar in some respects, in practise, to the "command objects" approach talked about in this thread.

Wearing my other hat - due to limited free time, some website changes take a while to implement from start to finish. Sometimes this can also be the case where I work, but not always. During this time the content is likely to be changed by the client a lot. In this instance, it's not really feasible to impose a no-change restriction on the site whilst I spend days and weeks implementing a change or new feature.

Taking those into consideration, I think what would work for me or the company, is some form of file-based change or altering commands or instructions which can be ran on a destination site, and have them made on there. Whether the ProcessWire UI logs or shows these in the UI or actually generates files  (as a configurable item) doesn't make that much difference to me and my workflow.

[ryan]

In case anyone is interested in trying out some of the things I was talking about in previous posts here, the latest dev branch has a field import/export function. You'll see it in the lower right corner of Setup > Fields. It enables you to copy and paste any fields across any PW installations. Locally, I also have this working for templates (with fieldgroups), though that part needs a little more work so it's not yet committed. I also have fields, templates and fieldgroups mirroring every change to JSON files, as an option that can be enabled for those that want to version these things with Git and what not. That part also isn't yet committed to dev, but will be soon. However, I figured the copy/paste function probably had the largest use potential. It makes migrating field changes (or creation of new fields) quite a simple task. Next up on the commits will be the same thing for templates (with fieldgroups). 

(note I didn't take these screenshots together, so they aren't referencing the same fields). 

[apeisa]

Wow - works really great. I like both things - will definitely start putting field&template changes under version control, but also love how this will simplify even the simplest scenarios: exporting & importing some features from one site to another (or from dev environment to live).

[Craig]

That looks awesome Ryan!

[Jonathan Lahijani]

Amazing!

[Nico Knoll]

Looks really really helpful!

[Soma]

I was working on a new fieldtype the last couple days and seen the export method and was wondering "what the hell is that?" Looks awesome!

[Soma]

I tried with a cache field and I get an error:

 TemplateFile: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'pw.field_searchindex' doesn't exist<pre>#0 /Applications/MAMP/htdocs/pw.ch/wire/modules/Fieldtype/FieldtypeCache.module(108): PDOStatement->execute() #1 /Applications/MAMP/htdocs/pw.ch/wire/modules/Fieldtype/FieldtypeCache.module(168): FieldtypeCache->getNumPagesCached(Object(Field)) #2 [internal function]: FieldtypeCache->___getConfigInputfields(Object(Field)) #3 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(359): call_user_func_array(Array, Array) #4 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(317): Wire->runHooks('getConfigInputf...', Array) #5 /Applications/MAMP/htdocs/pw.ch/wire/core/Fieldtype.php(182): Wire->__call('getConfigInputf...', Array) #6 /Applications/MAMP/htdocs/pw.ch/wire/core/Fieldtype.php(182): FieldtypeCache->getConfigInputfields(Object(Field)) #7 [internal function]: Fieldtype->___exportConfigData(Object(Field), Array) #8 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(359): call_user_func_array(Array, Array) #9 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(317): Wire->runHooks('exportConfigDat...', Array) #10 /Applications/MAMP/htdocs/pw.ch/wire/core/Field.php(180): Wire->__call('exportConfigDat...', Array) #11 /Applications/MAMP/htdocs/pw.ch/wire/core/Field.php(180): FieldtypeCache->exportConfigData(Object(Field), Array) #12 /Applications/MAMP/htdocs/pw.ch/wire/core/Field.php(217): Field->getExportData() #13 /Applications/MAMP/htdocs/pw.ch/wire/modules/Process/ProcessField/ProcessFieldExportImport.php(188): Field->setImportData(Array) #14 [internal function]: ProcessFieldExportImport->___buildImport() #15 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(359): call_user_func_array(Array, Array) #16 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(317): Wire->runHooks('buildImport', Array) #17 /Applications/MAMP/htdocs/pw.ch/wire/modules/Process/ProcessField/ProcessField.module(1008): Wire->__call('buildImport', Array) #18 /Applications/MAMP/htdocs/pw.ch/wire/modules/Process/ProcessField/ProcessField.module(1008): ProcessFieldExportImport->buildImport() #19 [internal function]: ProcessField->___executeImport() #20 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(359): call_user_func_array(Array, Array) #21 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(317): Wire->runHooks('executeImport', Array) #22 /Applications/MAMP/htdocs/pw.ch/wire/core/ProcessController.php(201): Wire->__call('executeImport', Array) #23 /Applications/MAMP/htdocs/pw.ch/wire/core/ProcessController.php(201): ProcessField->executeImport() #24 [internal function]: ProcessController->___execute() #25 /Applications/MAMP/htdocs/pw.ch/wire/core/Wire.php(359): call_user_func_array(Array, Array) #26

...

[ryan]

Soma, thanks for testing it out. I'm guessing there are a few Fieldtypes it's not going to work with yet (I've tested most, but not yet all). This whole system should be considered alpha at this stage, so the usual disclaimer applies: use it for play rather than production. 

In the case of the error message you got there, it looks like FIeldtypeCache has a bug where its attempting to access its DB table before the field exists. I think this is something that has to be fixed in FieldtyepCache rather than import/export. But I would like the export/import to capture those kinds of exceptions, so this'll be a good one to test with while I fix it. 

[Nico Knoll]

What about module configuration? Is it exported/imported, too?

[ryan]

The configuration as it applies to Fieldtype or Inputfield modules is exported/imported. If you are talking about other modules (like the configuration that's in Admin > Modules > Site or Core), then it's certainly feasible to do it the same way. Something to consider for the future. 

[Nico Knoll]

Yeah, I meant all of the modules (like Process modules or Markup modules).

[bcartier]

I really like where this is heading! I'd like to see it handle all Modules as well...   It could really improve our workflow.

[Nico Knoll]

Maybe you could even include auto recognition. Like calling the exported file "update.json" and put it into the main dir and the module recognizes it and asks you if you want to use this file for updating. Afterwards it deletes the file.

Would be that great! 

[SiNNuT]

Ryan,

I've been playing around with the  field export/import tool (dev branch, d.d. 11-8-2014 14:16:16 (UTC+0000).

I've only used it for exporting and importing new fields from one install to another, on a set of about 25 fields. Types used: float, text, textarea, integer and datetime.

It is already a huge time-saver! Recreating more than a couple fields fields manually is really time consuming and error-prone, and i also don't want to be doing this on the database level. So as far as i'm concerned a major improvement in (development) workflow.

One thing i noticed:

The created JSON always seems to be putting "required": 1 , even when the exported field is not required.

[ryan]

SiNNuT, glad you are liking this tool. I just committed an update that should fix this (I had an array_merge() with the wrong order of arguments). Actually, I was getting required as always empty (even if it was required) so the behavior I saw was a little different than what you mentioned. Can you confirm it fixes the issue?

[SiNNuT]

SiNNuT, glad you are liking this tool. I just committed an update that should fix this (I had an array_merge() with the wrong order of arguments). Actually, I was getting required as always empty (even if it was required) so the behavior I saw was a little different than what you mentioned. Can you confirm it fixes the issue?

I just pulled in the latest commits and tested again. The issue still is present: In my case the JSON always has "required": 1 no matter if the exported fields are not required.

On top of that another issue might be introduced: With a datetime field present in imported fields it now takes multiple commits and no matter how may times i press 'Commit Changes' the same screen reappears (although the setting in question seems to be saved just fine). See screenshot.  The Previously this wasn't the case.

[ryan]

I just pulled in the latest commits and tested again. The issue still is present: In my case the JSON always has "required": 1 no matter if the exported fields are not required.

I can't seem to duplicate this one. Can anyone else? I have a few work-in-progress files I've not yet committed, and don't think they should affect this, but that's my best guess. Can someone else confirm they are seeing "required: 1"  regardless of the field's required state?

On top of that another issue might be introduced: With a datetime field present in imported fields it now takes multiple commits and no matter how may times i press 'Commit Changes' the same screen reappears (although the setting in question seems to be saved just fine). See screenshot.  The Previously this wasn't the case.

Properties that begin with an underscore, like _dateInputFormat are for runtime use only. Previously it would blindly accept those. Now it remembers not to save them in the DB. I'll update the export function to exclude them as well. But since the property shouldn't be there in the first place, go to Fields > act_start_date > Alert. Check the box to delete that property and Save. Though if you don't see it listed as a property, you might also be able to solve this just by going to the field and saving it without doing anything else. 

[rajo]

I can't seem to duplicate this one. Can anyone else?

I can't. I just pulled the latest dev branch. I export a field that's not required and it shows required:"". I change its setting to required and it shows required:1.  I change it back and it show required:"".  IOW, it works for me as designed.

[SiNNuT]

Stop the presses! I must not have been thinking clearly yesterday. I pulled in the latest changes, updated the PW instance in question and went on to test it  on the wrong domain, which still had the old code I guess this is a good lesson to not have too many quite similarly named test domains.

So long story short: Latest dev has fixed the issue for me. As for the other one, this also seems to have to do with me exporting from an old install and importing that in a new (current dev) install. So everything is looking fine running the dev where the latest commit is '6ee2d96737fa64681fd3aefc65529ffca194f987'.

I'm gonna go away now and hide in shame

[Nico Knoll]

It's always the same confusion with live and dev versions

[Pete]

This is going to be really useful for me too - on larger sites I often remember to upload changes to template files but miss a critical change I made in one field or forgetting to add a field to a template.

Obviously it's not going to stop me being absent-minded from time to time, but it will help ensure I've got things configured the same over dev and live