"Continuous integration" of Field and Template changes

[mindplay.dk]

@Hari the idea of recording SQL statements and using that as a means of implementing migrations was already discussed earlier - it isn't viable. If you examine the queries that get executed, you will realize quickly why this approach doesn't really work. For one, template-names and field-names are not used in queries - the primary keys are. For another, some modules cause INSERT/DELETE/UPDATE statements that make changes to meta-data - these queries would be impossible to distinguish from queries that make changes to content.

I haven't made any progress on the module these past couple of months because I've been too busy, but the module is actually very far along. I haven't done any ProcessWire work recently, so I haven't had an occasion to get my head back in this project...

I still want to finish it, and it's actually not too far from usable.

[Hari KT]

@mindplay.dk

Thank you for the comments. Is your module available some where, so I can have a look ?

I noticed one gist. Or is there something else ?

 

[mindplay.dk]

Sorry, it is closed-source for now. It will probably be open-source eventually, but I have to figure out how to dual-license it...

[ryan]

Hi I was just wondering what you guys think about this

https://github.com/r...ssWire/pull/273

And you can use a tool like dbv or something similar .. Thoughts ?

Regardless of whether it would be useful in this instance or not, if it'd be useful to have this type of hook for anyone, I'd be happy to add it. Though rather than having the code in Database.php (now WireDatabasePDO).php it would instead be added as a hook that a module could monitor and take action when desired. 

I haven't made any progress on the module these past couple of months because I've been too busy, but the module is actually very far along.

I did get a preview of mindplay.dk's module and it's very nice. Rasmus I hope you get the time to finish and release it sometime, you always do great work.

For one, template-names and field-names are not used in queries - the primary keys are. 

You are right on that from a portability standpoint, as the IDs may not probably won't be the same between systems, so could never be relied upon. But I suppose the queries could always be altered to reference the name rather than the ID. 

For another, some modules cause INSERT/DELETE/UPDATE statements that make changes tometa-data - these queries would be impossible to distinguish from queries that make changes to content.

I'm not sure that I necessarily understand on this point or if it's something that can't be worked around, though you've obviously spent more time coding towards this need than I have. But just to help me understand, can you give an example of what you mean?

[mindplay.dk]

I've been working hard on this module this week, and so far, so good - it's now recording and can repeat most changes to Fields, Templates and Fieldgroups.

One remaining issue is foreign keys... some field types store Template IDs, Page IDs or other types of ID within the Field data.

For the known field types such as PageReference, I can handle those cases explicitly, but for third-party types, there is no saying what they're storing, where, how or when.

The only real solution I can think of is some kind of hook that permits other modules to support this module explicitly, by implementing methods that clean up foreign keys and replace them with names, and vice-versa... but that's not really elegant and the module won't work generally with every third-party field type out of the box.

Any thoughts?

[ryan]

The only ones I can think of that you would encounter in the core are template_id or parent_id. Those are used by Page reference fields, PageTable fields and possibly others. If you want to support things beyond that, I'm guessing you'd convert a template_id to a template name, and a parent_id to a page path? You are right that without knowing what the properties mean ahead of time, you'd probably need some interface that externalizes those values to something considered more portable. The thing is I'm not entirely sure that template name or page path would be entirely portable either? It might not always be preferable to the ID, and probably just depends on the situation. You'd instead need some kind of shared GUID that is independent of either system. That may not be realistic, so perhaps just a notice or confirmation during the migration process that certain properties would need to be double checked? 

[mindplay.dk]

Names should be reasonably safe - it's a marginal case where two developers both decide to add a template named "blogpost", and it won't work either way, since you can't have two templates with the same name in the first place, so whether or not a GUID or some other new identifier enables you to identify the referenced template, you're still stuck with a problem that cannot (and should not) be solved.

The system makes assertions for every possible type of migration, ahead of attempting to apply them - for example, "add template" will abort the migration process if a template with the same name already exists, etc... so it's already reasonably hardened against poorly coordinated changes.

It's not intended (or expected) or designed to deal with poorly coordinated changes - it's not a "version control system", it can't "diff and merge" database changes or anything of that sort. What should work though, is if two developers independently add non-conflicting fields and templates, after coordinating and planning their work properly, and you'd be able to merge all the changes two ways. That's really all I was expecting or hoping for. It will permit other changes, like re-ordering of fields in a template, but the last applied order "wins", there is no (worthwhile) way to merge those changes.

> I'm guessing you'd convert a template_id to a template name, and a parent_id to a page path

Exactly right

> You are right that without knowing what the properties mean ahead of time, you'd probably need some interface that externalizes those values to something considered more portable

Probably the simplest (perhaps the only?) solution is to add hooks, so that field types can replace foreign keys with names or paths, and vice-versa.

It makes me wish though, that field values containing e.g. template/page/field IDs were identifiable as such... this could be done in at least three different ways:

1. Foreign key values could be somehow identifiable as foreign key values themselves - e.g. using some kind of value-type or value-object, or a URI scheme of some kind, like for example "pw://template/123" rather than simply "123".
2. Foreign key values could be made globally unique, e.g. using GUIDs, so that when you encounter a GUID, you can go back to the ProcessWire API and ask for the resource with that GUID, whether it be a Template, Field, Page or some other PW type.
3. Adding some kind of "schema" to Page models - e.g. allowing you to define what you're storing; so you would still be able to store an integer of course, but there would be some way to define what datatype you're storing, e.g. Template ID or Field ID, etc.

I think every approach would have it's own advantages and drawbacks. A URI scheme would be very developer-friendly, since URIs would be easily readable in var_dump() etc... GUIDs would have advantages in terms of data portability in general, since any resource created on any system would be unique identifiable - this would help in the case of migration and import/export of any type of resource in general... Adding an actual schema to Page models might be the most complicated solution (for developers) and I'm not sure if it has any advantages over the other two.

Thoughts?

[mindplay.dk]

Here's a draft of a simple service adding resource URIs.

Just playing around, so don't take this too seriously

But the idea would be to replace numeric IDs everywhere in the model with URIs instead - and then, in Fields::get() and Templates::get() etc. you would convert or "sanitize" the ID to a numeric ID using something like $key = wire('resources')->id($key) ?: $key;

In addition, wire('resources')->get('...') would work for any type of resource - it uses the first portion of the URI path as the name of the resource collection, e.g. 'pages' or 'templates' etc. which can be resolved directly by just calling e.g. wire('templates')->get($id) etc.

Supporting URIs in page queries might be tricky, I didn't spend much time investigating that, but it doesn't look trivial.

Anyway, curious to hear what you think of the idea?

[rajo]

Resonating with this thread, the need for some continuous integration tools, and more, I've created a new topic to point to crude tools I just published on github. They go in a slightly different direction than where mindplay.dk is headed, but hopefully they help the cause.  See https://processwire.com/talk/topic/6779-continuous-integration-profile-management-etc-redux/

[djr]

I've been thinking about this issue lately also. I want the template/field metadata to be stored in files, not the database, since the metadata is closer to code than data. I think that using some sort of system to replay changes is a bit of a cop-out. I'd much prefer if no database changes were necessary. 

This means:

1. Template metadata has to be moved out from the templates table to files on disk. Easy enough.

2. Field metadata has to be moved out of the database. This is a bit trickier than the template metadata.

The data in the fields table could easily go in a file, but the issue is the field_<name> tables that are created for each field. Obviously, you can't just use a single generic table as it won't have the correct columns etc.

My first thought was that using a schemaless/"NoSQL" document database (e.g. mongo) would be a great solution to this problem. Just have a single pages collection, and store the field data as embedded documents. The field metadata can be moved out to a file, and since the database is schemaless, there is no issue with having the wrong database schema for the fields.

I have been playing with this idea over the past few days. It has a number of issues- it would be a lot of work to change the whole codebase to use a schemaless database; such database systems aren't as readily available as MySQL is; ... 

So, this morning it struck me that there was an easier way to accomplish the desired effect (without the need to change to a schemaless database system). The issue with putting all of the field metadata in files is that the field_<name> tables still need to be created with the correct schema.

But actually, do we really need these field_<name> tables? I don't think so.

Instead we can have a table for each fieldtype (created at install) which stores all fields of that type. The rows of these tables are linked to the row in the pages table.

This also means that fields do not have to be a global construct, but can be specific to each template (e.g. two templates could have fields with the same name but different type or other configuration difference). I feel that this is a more natural way to model the data.

I guess the schema would look a bit like this:

Any thoughts?

[rajo]

djr,

off the cuff, I'm not quite following where you're headed.  Are you discussing changing the pw model, or the need for tools to manipulate the metamodel?  Your 2 points about getting the template and field models to live in text files for source control sound like my proposal and my upgradewire tool, but thereafter I lose you.

I continue to be convinced that it's possible to use source code (php) to maintain site models and use PW's API to apply the model to a site (DB).

[teppo]

@djr: first of all, welcome to the forum! Very interesting first post you've got there

I may be the wrong person to comment on this, as I haven't had much time to think about these kind of changes (too busy building stuff on ProcessWire to spend time thinking what to change about it, I guess) but your idea made me wonder ..

1. how could selectors work properly with such a structure (not knowing for sure that a specific field behaves same in different places sounds like a problem) and
2. what kind of effect would it have to scalability and performance (I'd imagine this leading to tables with hundreds of thousands or millions of rows quite easily, which could become an issue considering overall speed, indexes etc.)

Current database structure works quite well in both respects, so this is definitely something that would have to be considered carefully. Other than that, I'm actually not entirely sure that we're on the same page here about some of the things you've mentioned in your post:

• You mentioned that this way fields wouldn't "have to be global constructs", yet I see that as a good thing. Reusable fields rather than page-specific ones.
• I'm not entirely sure that I'd prefer to have field schema on the disk, even if it makes duplicating that schema in another installation in some ways easier.

One of the key strengths about ProcessWire, in my use cases, is that I can very rapidly build new data structures with the admin tools it provides. How well would that be in line with having that structure as files on disk, I wonder? They're definitely not mutually exclusive things, but don't seem to fit that well together either (though perhaps it's just that I can't see it yet!)

Anyway, interesting idea and definitely a good thought provoker

[djr]

I continue to be convinced that it's possible to use source code (php) to maintain site models and use PW's API to apply the model to a site (DB).

It's definitely possible, but I'm not convinced it's the best way to do it.

off the cuff, I'm not quite following where you're headed.  Are you discussing changing the pw model, or the need for tools to manipulate the metamodel?  Your 2 points about getting the template and field models to live in text files for source control sound like my proposal and my upgradewire tool, but thereafter I lose you.

I was talking about changing the PW model.

Your system essentially functions like a database (schema) migration tool. What I'm suggesting is avoiding the need to change the database schema at all.

I'm imagining that we could create two files for each template: one is the actual template PHP code (as normal), and the other is a file containing the template and field settings that PW currently stores in the database.

A very rough example using JSON as the metadata format: https://gist.github.com/DavidJRobertson/2db2dc82107d2faacb0b

(quite crude and messy, this is essentially just a dump of all the relevant data from the default-site. Presumably some defaults could be assumed so not everything in that gist would need to be specified. Or perhaps another php file (with a custom extension could) be used to configure the template programmatically.)

If fields don't need their own table, but use a table dedicated to their fieldtype, we do not need to make any modifications to the database for this to work.

[rajo]

I was talking about changing the PW model.

That's what I thought but was afraid to believe  Then I ask the same question as teppo: Wouldn't it bog down under data volume?

[djr]

@teppo:

Ah, of course, selectors could be a problem. I'm not familiar enough with PW to know how this could work properly. And you're right, reusable fields are convenient. So I guess we could have global fields as it is in the current codebase. That would also minimize potential issues in migrating existing code which makes the assumption that fields are global, not local to each template. 

However, the fields could still be stored in the filesystem rather than in the database. That doesn't need to get in the way of the GUI tools in the admin interface to administer fields/templates. They could function as they do now, just storing the metadata in files instead of the database.

There doesn't even have to be any user interface change at all really, so the experience of rapidly building data structures can stay- just with some extra benefits: ease of code version control, collaboration, and deployment.

Your idea made me wonder (...) what kind of effect would it have to scalability and performance (I'd imagine this leading to tables with hundreds of thousands or millions of rows quite easily, which could become an issue considering overall speed, indexes etc.)

I'm not sure how it will perform compared to the current system. I wouldn't expect there to be any noticeable difference for small sites, but for very large datasets, I don't know. Might be worth testing.

[mindplay.dk]

I've been thinking about this issue lately also. I want the template/field metadata to be stored in files, not the database, since the metadata is closer to code than data. I think that using some sort of system to replay changes is a bit of a cop-out. I'd much prefer if no database changes were necessary. 

...

Any thoughts?

You're right, metadata does not belong in the database, and this approach is a bit of a "cop out" - I'm just trying to do the best I can under the circumstances.

I don't think redesigning the schema really solves the problem though - the metadata needs to be separated from the data, which means it needs to get out of the database; separation is the main point, structure is actually not really an issue.

What I would really like to see, is a transaction design, in which command objects are used consistently when making changes to the structural metadata - this would completely solve the problem, and make it very easy to build a migration system. Simply put, direct changes to metadata would be encapsulated and kept strictly internal - to make a change, you would instead submit command objects - for example, to change the name of a template, you would have to submit a ChangeTemplateCommand object, which the system would then serialize and store in sequential flat files, and of course serialize and store the updated template state.

To migrate a system forward, you would simply unserialize any command objects starting with the last applied numbered file and submit them again.

The trouble is, the existing system was not designed with a clear separation of data and metadata - the same problem as in pretty much any CMS I can think of...

[tehandyb]

I agree this is a cool idea. But what's wrong with using mysqldump to migrate the database and just copy the site over to production?

[djr]

I agree this is a cool idea. But what's wrong with using mysqldump to migrate the database and just copy the site over to production?

Okay so what happens if:

I take a copy of the production site (incl. database) and make some changes on my local machine. At the same time my client logs in and edits something, or the production database is otherwise altered. Then, I dump my db, and upload it and the files to the production server.

You have just overwritten your client's changes.

If you can ensure that they will not make any changes during the time between you taking a copy, and uploading the changed database, it works fine. Unfortunately I cannot make this guarantee.

Additionally, since the metadata is in the database, rather than files, it will not be tracked by your version control system (assuming you're using one). This also makes collaborating with other developers much harder than it needs to be.

[ryan]

At some point, I'll setup fields and templates to maintain mirror copies of files on the file system for those that want to version control them that way. They are in the database for very good reasons, but that doesn't mean that they can't be on the filesystem too. It's just that the database is the better default place for them to live and be maintained from for the vast majority. I don't consider writable files particularly trustworthy in the grand scheme, so the master source would always have to be the DB, but the mirror files could certainly be monitored, versioned and migrated at the developers discretion. I have no problem with that so long as we're not letting the file system lead. 

Something to add here that I think gets overlooked is that PW is not just a development tool. In the majority of cases, the development phase is just a very short blip in the overall timeline of the project developed with ProcessWire (whether a website, application or whatever it is). While we put a lot of focus on the development tool aspect of PW, the big design decisions are made for the entire lifecycle, not just the development phase. CMSs have to be written for large scale realities and diversity across hosting providers, varying amounts of security and expertise levels. While files here seem beneficial from one perspective, that does not translate to a net benefit in the larger context. In fact it translates to something rather undesirable in the larger context. 

Having important data in writable files on the file system is something you try to avoid with a CMS. While they are a necessary evil in some cases, if you can keep writable data in the DB, it's going to result in stronger security over a broad installation base and over a broad time period. These writable files are often the weakest link on shared hosting accounts. They can be perfectly secure, but there's little doubt the DB is safer. When it comes to data that needs to be editable, I consider the DB a much more trustworthy storage mechanism for important data across a large set of installations. I'm stating the diverse reality of our big picture context and not any individual's server. Some of us are running on servers where it would make no difference at all from a security aspect, but that's not something we can count on. 

Outside of the situations mentioned in this thread, I think most would not find it desirable to have the field and template data that disconnected from the content it represents. I can imagine some real headaches with schema getting disconnected from the data. When I backup my DB, I like to know that I've got everything necessary to put it back together without having to chase down multiple files with multiple tools, not to mention possibly files in different versions. I don't want to open the door to having having schema files that are not consistent with the data that they are supposed to represent. Data loss is always a possibility with schema changes and should always be accompanied by a confirmation. Automation by movement of schema in files (whether by git or SSH, FTP, etc.) is problematic for a lot of reasons.

The issue described about one person's changes overwriting another's may be a potential real world case, but is a non-issue for most of us because we don't migrate those kinds of changes at the DB level, nor do I recommend doing that. I understand that there are challenges for a team of developers having multiple versions of schema, or developers that want to migrate schema changes to a live server in an automated fashion rather than re-playing those changes themselves. I actually think less automation is ultimately a safer solution here, even if not as convenient for some. Though I'm still very enthusiastic about continuous integration projects and doing anything I can to support them. But I do not support moving storage away from the DB as our primary storage for these things. I understand using the file system may seem like a smart choice in certain contexts (and I don't disagree on them), but in the larger context it's not a wise choice. I'll do my best to find ways to mirror the data to files for those that might want this or may benefit from it. 

[djr]

Having important data in writable files on the file system is something you try to avoid with a CMS. While they are a necessary evil in some cases, if you can keep writable data in the DB, it's going to result in stronger security over a broad installation base and over a broad time period. These writable files are often the weakest link on shared hosting accounts. They can be perfectly secure, but there's little doubt the DB is safer. When it comes to data that needs to be editable, I consider the DB a much more trustworthy storage mechanism for important data across a large set of installations. I'm stating the diverse reality of our big picture context and not any individual's server. Some of us are running on servers where it would make no difference at all from a security aspect, but that's not something we can count on. 

I'm really not sure what the issue is with writeable files in regards to security. If the server is compromised in a way that would allow an attacker to modify these files, then you're already screwed as they could modify the code as well. Also in that case they would be able to access the database. The only issue I can see is files having incorrect permissions modes, which could be easily detected and warned about in the admin interface.

(Additionally, if I know that I will never want to make any metadata changes directly on the production site, I could set a configuration option (or something) to disable the editing interface for fields/templates, and then also make the files non-writeable. Though obviously this scenario won't apply to everyone.)

Outside of the situations mentioned in this thread, I think most would not find it desirable to have the field and template data that disconnected from the content it represents. I can imagine some real headaches with schema getting disconnected from the data. When I backup my DB, I like to know that I've got everything necessary to put it back together without having to chase down multiple files with multiple tools, not to mention possibly files in different versions. I don't want to open the door to having having schema files that are not consistent with the data that they are supposed to represent.

We already have an issue of inconsistency: the template PHP code may not be consistent with the data structure. If the template/field config is stored in files this issue disappears.

But with this model, if the metadata somehow becomes out of sync with the data, then the only real issue is going to be that some data might not be present (which would have been an issue anyway).

In fact it's arguable that it's not really possible for it to become out of sync, since the schema files are the authoritative source. If a field has been added, then everything just works, apart from obviously old pages won't have data for the new field. If a field is removed, not a problem, it's just not used any more. The data would still be there but orphaned (see bottom of this post re: cleanup task). If a field type is changed, the old data is again left in the old fieldtype table, and everything should just work.

Data loss is always a possibility with schema changes and should always be accompanied by a confirmation. Automation by movement of schema in files (whether by git or SSH, FTP, etc.) is problematic for a lot of reasons.

The whole idea was to avoid needing to change the database schema at all. Changing the field/template configuration to something incorrect could break the site until it's corrected, but would not cause any data loss. Since the files on disk would represent the desired state of the data, not the changes required to achieve that state, destructive actions would not be performed automatically. I imagine a cleanup task would be necessary to remove old data which is no longer required (e.g. when a field is removed). This would be run manually from the admin interface, so user confirmation would still be required.

[ryan]

I'm really not sure what the issue is with writeable files in regards to security. If the server is compromised in a way that would allow an attacker to modify these files, then you're already screwed as they could modify the code as well. 

You are not already screwed if your writable files aren't executable or don't represent files that can blow up the site. Obviously, the intention is that nobody else can write to the writable files, but the reality is when designing a system we have to plan for the worst. Your environment and my environment may be well secured, but I try to think about the bigger picture out there where there are a lot of insecure environments. Security is #1 when it comes to ProcessWire. 

 

 The only issue I can see is files having incorrect permissions modes, which could be easily detected and warned about in the admin interface.

 

 

What are secure permissions and what are not is dependent on the server and not something we can detect. A security best practice is to get everything important out of any writable files on the file system and not have any single writable files that could take down the whole system. Meaning, I think it's acceptable for cache files,  image and other independently served assets to exist as writable files, but the files we're talking about here are definitely out. Another reason we can't detect this stuff is because there are plenty of hosts running phpsuexec and the like where apache runs as the user and everything is writable by default. This tends to be okay, so long as there isn't also a copy of WordPress somewhere on the account. Basically, what are the right permissions for a given site comes down to the server, environment and host, and not ProcessWire. 

 

Also in that case they would be able to access the database.

 

If someone is able to write something to some file on your account, that does not imply they can access the DB. That "someone" is usually an automated script and not a real person, or it's one of your account neighbors on a shared webhost. One of the things that regularly falls in my lap (for better or worse) is to fix hacked WordPress sites. Rarely has the DB been touched, while most of the writable file system has to be quarantined. 

 

We already have an issue of inconsistency: the template PHP code may not be consistent with the data structure. If the template/field config is stored in files this issue disappears.

I disagree. There is a direct connection of field schema to the data represented in the DB. Template files are not coupled to fields like that because you are talking about your code, not the system. You have to make your changes independently regardless of where they are stored. Changing a field doesn't automatically change a template file. But changes to fields are an end-point in terms of how they reflect on the database. 

 

I've run out of time for today so have to stop. But I want to be clear that PW is not going to regress to storing this stuff on the file system. I recognize there is a benefit to some developers in having files they can version control or migrate independently. I personally would find the drawbacks to far outweigh the benefits (I have to support this project for a large audience), but understand why some find it desirable. Like I've mentioned, I would consider in the future mirroring the data to files that you can version control and selectively migrate in the admin. I believe that would bring the benefits of what's been discussed here, without the drawbacks.

[mvolke]

For all of you who want the model stored in files, maybe you should have a look into Bolt CMS. 

In Bolt you can configure your content types in YAML files and import them via a migration tool integrated into the CMS.

In my opinion this is by far the best approach. This enables...

• versioning of data structure
• easy deployment
• working in teams (because branching/merging in Git workflows is now possible)
• much faster editing of data structure (for professionals) than clicking through UIs

What I miss in this discussion is, what is the audience of users of Processwire?

Do we want to be attractive to professionals working for clients (I would count me to this group) 

or is Processwire just another CMS that will blow up its backend UI (like Joomla, Drupal, Wordpress) because it wants to be attractive to users without any programming skills?

My personal opinion is you cannot combine both groups in one system.

[rajo]

My personal opinion is you cannot combine both groups in one system.

I don't agree. I believe the simplicity of the PW backend metaphors is powerful, easy to understand, and easy to navigate for any level of user.  Additionally, it's possible to use the powerful API to bolt on a parallel source-file-based approach (see my earlier post about upgradewire).  I'm not done making it complete, but I think then you can have both ways, without disrupting what's in place, oh so zen, and yet accommodating of other development models.

I know it's far from complete and it's "different" than most proposed methods here which tend toward being integrated instead of on-the-side (i.e. modules and such, instead of stand-alone PHP), but I was expecting more discussion of my proposed approach here. A little vanity, maybe.  Seriously, what am I missing in my proposal which would leave the current integrity of the system in place and allow for alternative development style that meet the requirements this thread is about?

Speaking to YAML approaches, I thought it would be simpler and more functional to have an API like the one I made in upgradewire, where the model changes are expressed functionally instead of statically. It's about finding the middle way to expressivity, to have a lingo that's both declarative and functional.

[ryan]

Do we want to be attractive to professionals working for clients (I would count me to this group) 

or is Processwire just another CMS that will blow up its backend UI (like Joomla, Drupal, Wordpress) because it wants to be attractive to users without any programming skills?

Both. The goal with ProcessWire is to be attractive web designers/developers with programming skills and those without programming skills. ProcessWire is primarily a CMF, somewhere in between a framework and a CMS, but with the ability to be completely one or the other as needed. The admin UI is an application on top of the API. No action in ProcessWire actually requires the admin UI. The admin UI is there to help you. If you don't like that kind of help, you are likely in the minority, but ProcessWire still has you covered. However, if you want to work purely in code you should also look beyond CMF/CMS and towards dedicated frameworks like Laravel.

ProcessWire is fundamentally different from any of the CMSs you mentioned, and if you don't realize that yet then I'd encourage you to get to know the system better, and we're all here to help. While our system is fundamentally different, the web designer/developer side of our audiences are similar to those CMSs. Speaking outside of anything specific, we can't ignore what are likely our largest audiences. 

My personal opinion is you cannot combine both groups in one system.

Perhaps in your system. Whatever limitations your opinion is based on are not present in ProcessWire. I put together a proof-of-concept here when the request came up a few messages above and I can assure you it works quite well. If there's enough demand for it, I'll finish and migrate it to the core. Though if it's something that less than 30% of people will actually use, then it doesn't belong in the core. This is why I'm more enthusiastic about supporting specific-audience features like this by extending API calls and hooks that help other module developers build these tools. 

Even when tools like this exist in a stable state, I see their value as being for migration and versioning, not for configuration. I don't believe any developer would choose to configure the rather comprehensive aspects of templates and fields from a text file. ProcessWire is more powerful than the systems you've mentioned. The options available to configure for any given field/template are not predefined like they might be in other systems or frameworks. The options vary broadly depending on what Fieldtype and Inputfield is in use (all of which are themselves plugin modules that define their own options). Options can also be affected by other plugin modules. Dependencies are also in place, meaning the presence and requirements of some configuration options can depend on the values already present in other options. Further, some Fieldtypes can ride on top of others (like some of the ProFields), maintaining their own configuration along with those of their host. All of these things lend themselves extremely well to a user interface, and greatly benefit the developer in many ways, especially with regard to instruction and documentation. All of these things can be configured from the API, and could certainly be configured from a text file. But I would consider myself very lazy and remiss in my responsibilities if I expected people to use text files (YAML or otherwise) as the primarily method of configuration for this. I sure wouldn't be a happy web developer having to use that. On the other hand, if you are dealing with a system that is limited to predefined options that are self documenting, then I'm sure a text file for configuration is just fine. But there's little point in comparing ProcessWire to such systems, because ProcessWire provides a level of power that goes far beyond that, and far beyond what I think anyone would find desirable to configure from text files. We use a user interface here for exactly what a user interface is meant for.

[mindplay.dk]

> We use a user interface here for exactly what a user interface is meant for.

Agree - however...

> I would consider myself very lazy and remiss in my responsibilities if I expected people to use text files (YAML or otherwise) as the primarily method of configuratio

I don't think anyone has proposed or suggested that? We're proposing a supplement/alternative, not a replacement. At least I would never suggest that, and I don't think that's what rajo was trying to imply.

Ryan, do you have any thoughts on what I mentioned a couple of times earlier - changing the API internally to use a command pattern for changes to the data model? If every change to the data-model had to be submitted to a command processor in the form of an object, and those objects could be serialized, and assuming the command processor was hookable, that would make it trivial to implement all kinds of synchronization / change management / logging / recording modules.

The problem with manipulating schema directly, is you can't only tell whether something changed, not how it changed or why - you are already working around this fact by introducing things like change-tracking internally in the objects, which need to know what changes were made. For example, when a field gets renamed, there are methods to capture that change and work around it's side-effects.

When dealing with schema, I find it safer to have a centralized facility through which detailed records of change pass through - rather than manipulating individual aspects of the schema independently here and there. Template changes and Field changes, for example, are closely related because they are components of the same schema model - yet these changes are completely unrelated at the API level. A command facility could provide a central API point where all schema-related changes would be exposed, even future (and even third-party) changes would pass through it, so for example a synchronization module could work with future/third-party extensions to the data-model...