Version Control Wishlist for ProcessWire 4

[MoritzLost]

There's been a lot of talk recently about managing ProcessWire sites in version control systems (git in particular). Currently, this is quite difficult. The most important reason is that most of the site configuration is stored in the database, which you can't track in version control save for a database dump. But beyond that, there are many features and functions that don't work well with version control. To keep it short, I'll try to list everything I would like to see in terms of version control support in a potential ProcessWire version 4. For comparison, a similar system that does fully support version control is Craft CMS, which is also the inspiration for many of the approaches and changes listed below.

Goals and paradigms

Version control isn't self-serving, but enables several important development practices and workflows. This includes the following:

• Having a complete version history for a site, and being able to revert to a specific version or restore earlier versions of specific files and features.
• Being able to work asynchronously on a project with multiple people, without the need for excessive coordination or a lot of work to merge parallel development work.
• Having a single source of truth for the entire site (excluding content) in version control. This allows you to:
  • Set up automated deployment pipelines (dev -> staging -> production) with no manual steps required for a deployment.
  • Set up new instances of the site (additional staging environments, new live site if the server burns down, etc) with little to no manual effort.
  • Being able to switch between development branches and instantly sync up to the site state in that branch (especially useful for pull request reviews).

Everything on following wishlist will be in service to those goals.

The wishlist

Declarative config. This is the most important thing. ProcessWire needs a config schema that includes all the fields, templates and site settings of the site, alongside tools to import/export this config. This allows you to modify the site configuration in a development environment and instantly apply those changes to other environments. There's been some discussion about declarative config vs migrations recently, so I'll just quote my points on this debate:

Quote

 

YAML is preferable because it's declarative instead of imperative. This has a couple of side-benefits, like cleaner diff views in git, no formatting issues or different styles and no 'noise' in your commits (all only relevant if you have a git-based workflow with pull requests). But the big thing is that it makes it impossible to create environment-specific configuration, which is exactly what you don't want. If you embrace that the configuration is the source of truth for the entire site state (excluding content), you won't need this anyway. Take your example where you switch a field based on whether the languages module is installed - I would flag this in a PR and consider it an antipattern. Whether a site is multi-language or not should be part of the configuration. If it isn't there's no way to guarantee that the code which works in staging will also work in production, so at that point you're doing all the work for controlled deployments and version control but not getting the benefits.

Another downside of PHP is that it's onedirectional by default. With YAML, if a deployment fails, you can just roll back to the earlier version and apply the configuration of that version. With PHP, this may work if the PHP migration is just one single $rm->migrate call with an array of configuration (so basically it is a declarative config). But you have no guarantees that it will, and if you have any logic in your migration that depends on the previous state of the site to migrate to a new state, this migration is irreversible.

Migrations do have their place - if you really need to perform some logic, like moving content from one field or format to another. But besides that, declarative configuration files are preferable.

 

More on that in this thread. For reference, Craft provides this in the form of the project config.

Split ProcessWire into a starter project and a Composer dependency. ProcessWire is installed the old-fashioned way, by downloading a zip file and unzipping it in the web root. This leaves you with a problem – either include all the core files in your version control, which muddies your version history, or set up custom hooks or scripts to repeat this step during deployments. In addition, a default ProcessWire installation will include lots of directories and files where it's not clear if they belong to the core or are intended to be modified and included in your site repository. This can be solved by splitting ProcessWire into two repositories. One is the actual ProcessWire core, which can be installed as a dependency using Composer. The other is a starter project that only contains a site skeleton and is intended as a starter point for your site's directory. Other site profiles can become alternative starter projects that you can choose from. Craft comes as a starter project (craftcms/cms) which only includes the skeleton for the most important directories and files and installs the actual CMS as a dependency (craftcms/craft).

Better directory structures. The current directory structure doesn't separate core files and site files well, and for many directories it's not clear whether you want them included in version control. For example, the site/assets/files folder contains uploads for assets, so you want to exclude them from version control, right? The problem is that for some file go into that folder that you do want to track, so you have to manage that manually. Translation files, for example, go into the assets folder, but you probably want those in version control. Another example are modules – custom site-specific modules go into the same directory as contributed community modules. You want the former in version control, but not the latter.

Make modules installable with Composer. Pretty self-explanatory – currently you can only install modules by, again, unzipping folders manually or through the module interface. You can't install modules automatically during deployment (at least not without some custom scripting and workarounds). Making those installable with Composer would solve that issue. This goes for Pro modules as well.

Better distinction between content and configuration. To support deployments that update a site configuration to a particular state while leaving content untouched, you need a clear distinction between config and content. This is important if you are developing in your local dev environment while the live site is constantly updated with content. If the site has user-generated content, you can't just import the development database from three weeks ago and call it a day. But for many settings and features it's not clear whether they are content or config. For example, what about multi-language sites – are the available sites content or config? Languages are just pages under the hood, so it looks like they're content – but there are some good arguments to be made for tracking the available languages in version control. Another example are page reference fields. Those allow you to select default pages and limit selectable pages to a specific parent page – but now you have content (specific pages) in your configuration. Those pages might not exist on the target system, so this feature might break unexpectedly in a different environment. For comparison, Craft's project config explicitly defines everything that goes into the config – everything else is content and won't be included in version control or deployments.

Provide CLI tools for deployments. Many of the operations you need for a deployment can only be performed through the interface, so it doesn't work with fully automated deployments. To address this, ProcessWire needs to provide a CLI tool to automate those operations. For example, installing modules, synching up the project config (see the first point) and clearing caches need to be automatable. Craft provides a CLI with those and many other commands.

Better integration with standard development tools. ProcessWire uses a lot of custom solutions instead of standard tools and practices. For example, it provides a custom interface and installation method for modules, which could be simplified a lot by just using Composer for module installations. By the way, it is possible to provide a user interface for Composer operations, Craft does exactly that. But I don't think this level of polish is necessary. Simply using standard tools and practices that are compatible with version control by default will reduce development time for those features and at the same time be easier to work with for developers.

Support for environment variables. You don't want passwords and other credentials tracked in version control, and you want to be able to distinguish between environments (dev, staging, production) to toggle some features or change some settings. The standard way to solve this are environment variables. This way, you keep database configuration, SMTP servers, passwords to external services and so on out of version control and provide custom values for those depending on the environment. Craft, for example, allows you to change all configuration values based on the environment, and disables any configuration changes in production environments by default.

Webroot in a subdirectory. This is a minor point and admittedly only tangentially related, but it's a no-brainer once all the other changes are made. The project root should not be the webroot because that makes all project files publicly accessible by default. To fix this, ProcessWire uses a lot of .htaccess rules and .htaccess files in subdirectories, but that only works if you're using Apache. A lot of modern server management tools use other servers, like nginx or Caddy, so those require additional work to secure a ProcessWire installation. The fix here is to point the webroot to a subdirectory (like web/ or public/) and only put files that you want to be publicly accessible in there.

-------

All those changes constitute a major shift in mindset and approach, which is why I think they would need to go into a future ProcessWire v4. To be honest, at this point I don't think it's likely to happen, since Ryan has repeatedly signaled that he doesn't see the value in those changes. While there are promising attempts to solve this from the community, true support for the goals outlined above can only come from the core. As long as that doesn't happen, ProcessWire won't be an option for any projects that require more than one or two people working on it or that need continuous development and deployments – at least for me. Not as long as there are other alternatives, like Craft CMS, where I get all those benefits out of the box.

[Ivan Gretsky]

On 3/31/2022 at 5:05 PM, MoritzLost said:

As long as that doesn't happen, ProcessWire won't be an option for any projects that require more than one or two people working on it or that need continuous development and deployments

Totally agree. ProcessWire is so amazing I wish I could grow with it in larger scale projects. I am sure that @MoritzLost's proposals could bring it much closer to this task. We should at least discuss that. Maybe, there could be a not-so-hard way to achieve that. Most of the configs are json already, so they could be stored in files instead. Moving to a better composer integration will bring us closer to what is considered mainstream php development nowadays (many PW gurus already tend to provide composer-based installation facilities). We do not want to sacrifice the PW way of doing things, but maybe now we can marry it with composer and git based workflow and move it to new heights.

What do you think? Maybe it is time to start planning ProcessWire 4?!

[bernhard]

On 4/1/2022 at 7:39 PM, Ivan Gretsky said:

On 3/31/2022 at 4:05 PM, MoritzLost said:

As long as that doesn't happen, ProcessWire won't be an option for any projects that require more than one or two people working on it or that need continuous development and deployments

Totally agree. ProcessWire is so amazing I wish I could grow with it in larger scale projects.

Totally disagree. At work we are 3 devs at the moment working on the same projects all the time and we have fully automated deployments both to staging and production. And it is easy to review and merge pull requests. And I've shared all that is necessary with the community. For free. And so has @LostKobrakai

 @MoritzLost Please stop stating your points in a manner that discredits the work of others or the way how others are working. Thank you.

On 3/31/2022 at 4:05 PM, MoritzLost said:

Provide CLI tools for deployments. Many of the operations you need for a deployment can only be performed through the interface, so it doesn't work with fully automated deployments. To address this, ProcessWire needs to provide a CLI tool to automate those operations. For example, installing modules, synching up the project config (see the first point) and clearing caches need to be automatable. Craft provides a CLI with those and many other commands.

We have wire shell. Why don't you take up this project? You are obviously in need of such tools. Why do you keep bashing around and do not help improving the tools that we already have or help to make new tools being created and - also important - being shared with the community?

I like that you bring up all those points, but I really don't like the way you are doing this. Imagine someone evaluating ProcessWire and somehow finding your thread or reading it by coincidence. Even if that person will likely not read it with the same emotions that I do he/she will still get an - in my opinion - wrong picture.

I think it is in the interest of all of us to promote the things we have and to encourage everybody to take part.

[MarkE]

I realise that the original post here arose from the context of a comparison of Craft and PW. While it is sometimes helpful to look at other CMSs for inspiration, my fear is that, in this case it clouds the essential point:

On 3/31/2022 at 3:05 PM, MoritzLost said:

There's been a lot of talk recently about managing ProcessWire sites in version control systems (git in particular). Currently, this is quite difficult. The most important reason is that most of the site configuration is stored in the database, which you can't track in version control save for a database dump.

The 'lot of talk' refers to 

where, indeed, there was a lot of useful discussion of this topic but, unfortunately, not a lot of progress. As I suggested in that discussion:

On 2/4/2022 at 12:12 AM, MarkE said:

As to a way forward, I think a collaborative development of requirements and spec would help and then some agreement on who and how to build it. I also really think that a contribution to this discussion from @ryan before proceeding would be most helpful.

@MoritzLost's wishlist is definitely a contribution to such a collaborative development and I would agree with much of what is said. My chief concern is that it is very aspirational (and also reflective of personal preferences) such that:

On 3/31/2022 at 3:05 PM, MoritzLost said:

To be honest, at this point I don't think it's likely to happen

I think if the aspirations were more limited (at least initially), there might be a better chance of a result. Before reviewing the 'wish list', I would like to firstly consider the range of views then, finally, look at the current state and the way forward.

Viewpoints

Both @MoritzLost and I refer to @ryan's views on the topic of migrations and version control, but I do not recall seeing those expressed specifically anywhere. I also note that, in the other thread, @horst gave a "-10" to my suggestion that it was a really important topic. If they do not think it is important, I would really like to hear why.

My viewpoint: I am not a tech wizard and my needs are quite simple; however, I have built a few sites in PW that make extensive use of its capability as a app-building framework rather than just a simple CMS. It is ideally suited to this purpose... ...except when you need to upgrade the live site. It is impossible to copy the live DB, change and test that then upload, without making it unavailable. Making anything but simple changes to fields and templates carries a high risk of human error.

Wishlist

As regards @MoritzLost's wishes:

On 3/31/2022 at 3:05 PM, MoritzLost said:

Declarative config. This is the most important thing.

Absolutely.

On 3/31/2022 at 3:05 PM, MoritzLost said:

Better distinction between content and configuration

That would be helpful.

The other wishes in the list are either 'nice to haves' or personal preferences (IMHO).

I would add one other 'wish' at this point:

"Ability to implement on an already-existing site."

EDIT: And another 'Roll-back capability'

Current state

At the moment the core provides practically nothing. There are some export features but they are incomplete and buggy in places.

@bernhard's RockMigrations module is a well-developed solution to the problem, but it is not really declarative and I think it is difficult to implement on existing sites. It is fine if you want to code rather than use the PW Admin UI and if you are doing everything the 'RockMigrations way' from scratch for a site, but I think it sits a little uncomfortably with the PW way of doing things.

For these reasons, I developed 'ProcessDbMigrate' as a POC (This is extensively documented here). In fact it rather exceeded my expectations and achieves the following:

1. To allow development to take place (in a separate environment on a copy of the live database, or on a test database with the same structure) using the Admin UI. When finished, to permit a declarative approach to defining the migration and implementing it (again in the UI).
2. To allow testing of the migration in a test environment on a copy of the live database.
3. To allow roll-back of a migration if installation causes problems (ideally while testing rather than after implementation!).
4. To provide a record of changes applied.
5. Optionally, if changes are made directly on the live system (presumably simple, low-risk mods – although not best practice), to allow reverse migration to the development system in a similar fashion.
6. To provide a database comparison tool which can (if wished) be used to create a draft migration for the differences between the databases.

I have no pride of authorship here (and I am sure some may think my coding is a bit hacky), but it is a fairly thorough attempt to address the wishes expressed above. It is declarative (using JSON rather than YAML). It deals with the content vs configuration issue by defining a 'scope' for every migration - in other words, it can be used when the distinction between content and configuration is a bit fluid (which is always a potential issue in any CMS). It can be simply implemented on any existing site by installing the module.

However, ProcessDbMigrate does have some drawbacks. 

• It introduces quite a few new templates, fields and admin pages which could confuse a user (although they are flagged as 'system').
• It doesn't completely and accurately model all possible fields and templates so it does need to be tested on the chosen site before implementing. This is partly because it is 'POC' and partly because I couldn't find a definitive model.
• I know for certain that it does not cope with nested repeater fields.
• EDIT: plus if you use it to compare a whole database it will probably crash by exceeding memory limits - however, it is more reliable when scoped to genuine configuration matters.

Way forward

As I said earlier, I think the way forward should be collaborative. I also think that a module is more practicable, at least initially, than expecting a massive core upgrade; it is also more workable for a collaborative approach.

I think that a specially moderated thread might be a good start, where the moderator is reasonably unpinionated but sufficiently experienced. 

Initially, views should be gathered (via a poll? - perhaps @teppo could facilitate this?) as to the need for a migration/version control solution. Then some ranking of the 'wish list' items followed by a selection of a 'doable' set of requirements. The existing solutions could then be compared with this list to assess whether any of them go some way towards meeting the requirements and might be adapted or incorporated (either as code or as a 'leg-up' on the specification).

Alongside this, we need a complete and accurate published data model of fields, templates and pages. Futhermore, this would need to be kept up-to-date for core changes. Existing modules (particularly Fieldtypes) would need to be reviewed for compliance. @ryan's help with this would be essential.

No doubt contributors to the thread will have their own views on how to progress things, but a willingness to make tangible contributions would be helpful!

Conclusion

Perhaps I have missed the point entirely, in which case, please explain, but my fear is that this discussion will just remain a discussion and it will be left to individuals with the motivation to provide their own solutions which inevitably will reflect their own personal preferences and needs.

[bernhard]

Thx for your post @MarkE and sorry for not mentioning your module in the other thread ?

9 hours ago, MarkE said:

On 3/31/2022 at 4:05 PM, MoritzLost said:

Declarative config. This is the most important thing.

Absolutely.

9 hours ago, MarkE said:

@bernhard's RockMigrations module is a well-developed solution to the problem, but it is not really declarative and I think it is difficult to implement on existing sites. It is fine if you want to code rather than use the PW Admin UI and if you are doing everything the 'RockMigrations way' from scratch for a site, but I think it sits a little uncomfortably with the PW way of doing things.

1) Could you (both) please explain what you mean by "declarative config". In detail.

2) Could you please explain what you mean by "RockMigrations is not really declarative". (Connected to 1)

3) Could you please explain why you think "RockMigrations is difficult to implement on existing sites".

4) Could you please explain why you think "it is fine if you want to code rather than use the PW Admin UI (*) and if you are doing everything the 'RockMigrations way' from scratch for a site, but I think it sits a little uncomfortably with the PW way of doing things."

*) I can somewhat agree on that until here (though that's the whole point of the topic: Having the setup/config in files, not in mouseclicks...) but the rest could not be further from the truth. Sorry. That's why I think the discussion was not really helpful. It was full of opinionated assumptions and expectations without any noticable efforts in doing research in what we already have (and we have tools for that since 2016!), which concepts of the available tools work well, which don't, which could work differently (eg. having migration config in YAML rather than in PHP which seemed to be a huge thing for some whereas it absolutely means nothing to me. And: Which would be such an easy thing to implement into RockMigrations if need be...).

I'll show why I think your statements are absolutely wrong in my opinion, but I'd be happy to hear your explanations first to get a better feeling of how RockMigrations is being seen by you or maybe also others, as I think that would be valuable feedback. Thank you ? 

[MarkE]

Thanks for the comments @bernhard. I'll do my best to reply. If, along the way, I demonstrate a misunderstanding about RockMigrations, please accept my apologies and help me to understand better - I have tried to use it but it didn't do what I wanted and I found it very time-consuming to try and get it right (admittedly this was some time ago and it has no doubt developed quite a bit since then). Equally, I would be grateful if you could try out ProcessDbMigrate, or at least read through the help file, and comment on the various pros and cons in order to advance the whole subject.

14 minutes ago, bernhard said:

1) Could you (both) please explain what you mean by "declarative config". In detail.

21 minutes ago, bernhard said:

Could you please explain what you mean by "RockMigrations is not really declarative".

A fair request. At its simplest, you could say that it is a (complete or, at least, sufficient) specification of the current state of the entity(ies) in question. Arguably, RockMigrations fits this definition, if you completely specify the entity. However that is time-consuming and it is simpler to just specify the changes (which is not fully declarative as it assumes a certain prior state). With ProcessDbMigrate, I was attempting to produce a declarative definition automatically, rather than having to code it. That might not be enough 'detail' but we can discuss further.

21 minutes ago, bernhard said:

Could you please explain why you think "RockMigrations is difficult to implement on existing sites"

Related to the above: to implement it in a declarative way required me to specify every field and template in full in code, by hand (at least for the migration scope in question). I had already done this via the UI and just wanted to produce the spec automatically from the database.

25 minutes ago, bernhard said:

Could you please explain why you think "it is fine if you want to code rather than use the PW Admin UI (*) and if you are doing everything the 'RockMigrations way' from scratch for a site, but I think it sits a little uncomfortably with the PW way of doing things."

Again, related to the above. There is not much point using the Admin UI to specify the fields etc. if you are going to do it via the API anyway. Many people like the UI and are less proficient at coding (typing, remembering all the structures and formats etc., even if aided by the IDE). Allowing specification via the UI does not mean that 'mouseclicks' need to be recorded - just that the config files need to be generated from the resulting database automatically (or at least programmatically).

31 minutes ago, bernhard said:

That's why I think the discussion was not really helpful. It was full of opinionated assumptions and expectations without any noticable efforts in doing research in what we already have

I hope you were referring to the OP here, not my post, as I did research all the prior migrations tools (even using some of the working bits of the core tools) and put a lot of work into my own solution.

33 minutes ago, bernhard said:

which concepts of the available tools work well, which don't, which could work differently (eg. having migration config in YAML rather than in PHP which seemed to be a huge thing for some whereas it absolutely means nothing to me. And: Which would be such an easy thing to implement into RockMigrations if need be...).

I quite agree. That's why I was suggesting some way of agreeing requirements and then reviewing the best way forward, which may be by adaptation of an existing tool. As regards YAML vs JSON vs PHP, I agree that it is not such a huge thing. After all, the PHP is just an array specification. However, I plumped for JSON as it seemed easier when I was wanting to automatically generate specs from the database. FWIW, my original intention was to generate RockMigrations-style PHP from the database and leverage your existing work, but I struggled with that (I did say I'm not a great tech...). That might still be a good way to go ?

40 minutes ago, bernhard said:

I'll show why I think your statements are absolutely wrong in my opinion, but I'd be happy to hear your explanations first to get a better feeling of how RockMigrations is being seen by you or maybe also others, as I think that would be valuable feedback.

In doing so, could you please also address the roll-back issue (added as an edit to my first post). Other requirements that ProcessDbMigrate meets which I think RockMigrations currently does not include database comparison, preview and audit trail. Please read the help file to get a full picture of what it does before commenting on pros and cons.

[bernhard]

Thx @MarkE I appreciate that! I'll need more time for an elaborate answer, but I have one quick point and one quick question:

2 hours ago, MarkE said:

I have tried to use it but it didn't do what I wanted and I found it very time-consuming to try and get it right (admittedly this was some time ago and it has no doubt developed quite a bit since then)

I think the point is not so much if RM has evolved or not. It's more about getting used to a new concept. The same would be true for your module even if there is a UI (which might make it easier, I agree). But still it's more time consuming in the beginning compared to working the way one has always been working...

2 hours ago, MarkE said:

However that is time-consuming and it is simpler to just specify the changes (which is not fully declarative as it assumes a certain prior state). With ProcessDbMigrate, I was attempting to produce a declarative definition automatically, rather than having to code it. That might not be enough 'detail' but we can discuss further.

Did you see this post? https://processwire.com/talk/topic/26565-share-your-pw-dev-setups-tools/?do=findComment&comment=220542

It took me around 20min to implement that. I'm still not a fan of this approach. Even more, I don't think that it will work. At least not for more complex projects and that's exactly the kind of projects where migrations come into play and what RockMigrations was built for... But - and that seemed to be ignored / not understood / stated wrong in several situations - RM does not have to be used in that "complicated" way. You have to write "code", yes. Actually it's php arrays of key-value-pairs... But I can understand that this might look complicated... I just got an idea ?

Before I answer the other parts... As many of those points have already been discussed. Did you see all my comments here? https://processwire.com/talk/topic/26565-share-your-pw-dev-setups-tools/?do=findComment&comment=220484 

 

 

 

[MarkE]

10 minutes ago, bernhard said:

Did you see this post? https://processwire.com/talk/topic/26565-share-your-pw-dev-setups-tools/?do=findComment&comment=220542

Yes. I was very impressed, but I didn't really understand the context (was this just a demo or a real working tool) and I wasn't clear what your plans were for it (to incorporate in RockMigrations somehow?).

12 minutes ago, bernhard said:

Even more, I don't think that it will work. At least not for more complex projects and that's exactly the kind of projects where migrations come into play and what RockMigrations was built for...

Because? It does seem to be very interactive and maybe that causes problems. With ProcessDbMigrate I very much opted for batch creation of entity specs, which is easier to manage.

14 minutes ago, bernhard said:

Did you see all my comments here? https://processwire.com/talk/topic/26565-share-your-pw-dev-setups-tools/?do=findComment&comment=220484 

 

Yes, I did, but I must confess to not having thought through all the comments made there. The intial part of the post clearly shows implementation on an existing site by just specifying the changes, not the whole description - hence my comment about 'not fully declarative as it assumes a prior state'. I will re-read it more thoroughly.

[MarkE]

OK @bernhard, I have now re read https://processwire.com/talk/topic/26565-share-your-pw-dev-setups-tools/?do=findComment&comment=220484 and haven't got much to add to my previous comment. I think a lot boils down to the target 'market' for the migration tool. PW actually has a rather broad user base - from those who want something just a bit more flexible than WP at one end to those who would be equally happy with a plain PHP framework. I have used both WP and CodeIgniter in the past and find PW way better than both. I am not a full-time professional developer - very much a part-time amateur on a few projects. I also hack around in Python from time to time. Consequently I do not instantly remember all the structures, attribute names etc. - the UI helps with all of that.

My view is that the migration tool (or tools) should cater for the full range of users and not just for the pro developer. My reason for this is (similarly to @MoritzLost) that PW without migrations is not a great solution for anything other than simple 'build and forget' websites which (forgive me ?) you may as well do in WP. Part-time or more amateur PW users need to maintain and upgrade sites just like full-time pros, but less frequently. You say in the post referenced above that it is much quicker to type the code than use the UI. I am sure that is true for you and your team, but most certainly not for everyone. For example, creating a new field, you enter 'type' => 'text' - well, that's easy, but what are all the other fieldtypes called? I would need to look up their names - the UI gives me a selection drop-down. Have I entered all the necesaary attributes for my new field? I would need some sort of checklist to consider; in the UI it is all presented to you. The IDE helps with methods, for sure, but not with strings, and not with completeness.

I appreciate it must be difficult to put yourself in the shoes of those less capable and experienced than you, but I think that is required to understand the needs of the wider user group. (I must admit that I don't understand @MoritzLost's problems with RockMigrations as I would have placed him in the super-pro category with you). You can take the view that only the super-pro group is important, but I think that only serves to marginalise PW. In my view, all users (other than those building very simple websites, for which PW is arguably overkill) need a solution to the migration/version issue even if they don't know it yet.

[bernhard]

On 4/4/2022 at 3:52 PM, MarkE said:

You say in the post referenced above that it is much quicker to type the code than use the UI. I am sure that is true for you and your team, but most certainly not for everyone. For example, creating a new field, you enter 'type' => 'text' - well, that's easy, but what are all the other fieldtypes called? I would need to look up their names - the UI gives me a selection drop-down. Have I entered all the necesaary attributes for my new field? I would need some sort of checklist to consider; in the UI it is all presented to you. The IDE helps with methods, for sure, but not with strings, and not with completeness.

Basically this was just a proof of concept to show that I could easily add a feature like this to RockMigrations and that this should in my opinion not be the point of the discussion... It was also there to show that it is just another way of making it easier for beginners to see a list of all the necessary or available properties of a field or template etc., but more on that later.

On 4/4/2022 at 2:16 PM, MarkE said:

On 4/4/2022 at 1:59 PM, bernhard said:

Even more, I don't think that it will work. At least not for more complex projects and that's exactly the kind of projects where migrations come into play and what RockMigrations was built for...

Because? It does seem to be very interactive and maybe that causes problems. With ProcessDbMigrate I very much opted for batch creation of entity specs, which is easier to manage.

Because - as far as I understood - the way it works makes it impossible to build reusable components which is a crucial part of being more productive and also increase quality. If you store migrations in a central place you lock the usefulness of that migration to the project. If you make it possible to store migrations where they logically belong (eg in a PHP class - or in the PW world in a custom page class), then you can create reusable components that you can move from project to project.

Is that more complicated? Yes. Do you have to do that? No! You don't have to build modules and you don't have to write migrations in a component-based way. If you don't care you can simply put everything in /site/migrate.php

On 4/4/2022 at 3:52 PM, MarkE said:

You say in the post referenced above that it is much quicker to type the code than use the UI. I am sure that is true for you and your team, but most certainly not for everyone. For example, creating a new field, you enter 'type' => 'text' - well, that's easy, but what are all the other fieldtypes called? I would need to look up their names - the UI gives me a selection drop-down. Have I entered all the necesaary attributes for my new field? I would need some sort of checklist to consider; in the UI it is all presented to you. The IDE helps with methods, for sure, but not with strings, and not with completeness.

This quotation is incorrect and unfair. If we are talking about this paragraph?

Quote

How long would it take you to delete those two fields manually? Making sure that all checks are met ("you can't delete this field... it is used by XX templates..." *#*&!), going back to the templates editor, removing the field first, then deleting the field, then going back to the home template and setting the title field back to 100%... I don't know the exact workflow - I haven't done that for a long time now ? Yeah and we are not even talking about doing those changes in a team... Have fun ? 

I was just demonstrating here that there ARE situations where doing things by hand is slower. I've never said that using RM is quicker all the time. I have to lookup things myself now and then. My point is that if you take the whole picture into account this equation changes (as in this example doing the same changes somewhere else - either in a team or on another setup).

On 4/4/2022 at 3:52 PM, MarkE said:

I appreciate it must be difficult to put yourself in the shoes of those less capable and experienced than you, but I think that is required to understand the needs of the wider user group. (I must admit that I don't understand @MoritzLost's problems with RockMigrations as I would have placed him in the super-pro category with you). You can take the view that only the super-pro group is important, but I think that only serves to marginalise PW. In my view, all users (other than those building very simple websites, for which PW is arguably overkill) need a solution to the migration/version issue even if they don't know it yet.

That part is half unfair and half ok.

The part that is ok is that I care more about my needs than I care about the needs of others. I hope you can understand that I am not keen on building a GUI based migrations tool when I know that I will not need it myself? Actually I do care about this much more than I should, but that's another topic... 

The unfair part is that I DO know how it feels to be overwhelmed by all this migrations stuff. I have been there when I first saw Lostkobrakais migrations module and I've had the same feelings (I guess) when he said "once you get used to it you'll be quicker writing code than doing things by hand". And I've heard the same last week when trying to understand the **** CMS where the core dev's show the product and nothing is as easy as they call it... And I've already taken the "feedback" and made the new version of RockMigrations MUCH easier to use and much easier to get started.

On 4/4/2022 at 3:52 PM, MarkE said:

My view is that the migration tool (or tools) should cater for the full range of users and not just for the pro developer.

Fine. Let's do that. I'm not against it. My point is that RockMigrations can serve both needs but with your approach I don't see a way to do this. So it's a little hard for me to read all those discussions and ideas and see (guess) where they would (could) lead to because I've been there...

On 4/4/2022 at 3:52 PM, MarkE said:

For example, creating a new field, you enter 'type' => 'text' - well, that's easy, but what are all the other fieldtypes called? I would need to look up their names - the UI gives me a selection drop-down. Have I entered all the necesaary attributes for my new field? I would need some sort of checklist to consider; in the UI it is all presented to you. The IDE helps with methods, for sure, but not with strings, and not with completeness.

I understand these concerns. They are easy to solve:

On 4/4/2022 at 3:52 PM, MarkE said:

For example, creating a new field, you enter 'type' => 'text' - well, that's easy, but what are all the other fieldtypes called? I would need to look up their names - the UI gives me a selection drop-down.

Your IDE gives you a dropdown as well - in VSCode its CMD+P (or similar on Windows):

On 4/4/2022 at 3:52 PM, MarkE said:

Have I entered all the necesaary attributes for my new field?

Most fields will just work with only the "type" attribute. For others I've started to build snippets as this is an issue that I have myself (so much for the topic putting myself into the shoes of less capable - I'm not better than you ? ?

On 4/4/2022 at 3:52 PM, MarkE said:

in the UI it is all presented to you. The IDE helps with methods, for sure, but not with strings, and not with completeness.

While the gif above shows that this is just another wrong statement based on false assumptions I want to add two more things. It's definitely not easy to remember all the properties of a field or of a template. But you don't have to! What if you wanted to change this field of a template?

First option: TracyDebugger (works always):

If that is too complicated there's also another option most of the time:

And finally:

 

Can I imagine that not everybody knows that? Yes. Why is that not in the readme? You'll get a similar answer to Ryan's related to the PW docs: I'm working on that alone. I'm not getting paid for anything here. And - and that's a huge difference to PW - I don't even know if anybody cares about migrations or RockMigrations in general. For a long time I thought nobody cares. But those threads show me that a lot of people actually do care about those topics. But still they seem to know nothing about RockMigrations.

It's as if I'd create a thread "E-Commerce Wishlist for ProcessWire" without mentioning @apeisa or @kongondo 's Padloper and without doing my homework to research what their module does, where it has limitations, asking why they built it the way they built it etc... If that had happened the whole discussion would have been on a totally different level (at least for me) and the several arguments that have been correct (like for example the fact that PW stores language files in /site/assets/files) could have been communicated to @ryan and I'm sure he'd have been supportive to do whatever he can to make the core serve our needs (even if he does not yet have a migration/git based workflow). Amen ? 

And finally to quote @Robin S from the other thread because I created the video initially for you @MarkE before you wrote your follow-up:

On 1/8/2022 at 5:17 AM, Robin S said:

On 1/7/2022 at 4:40 PM, MoritzLost said:

To keep track of schema changes, Craft saves all fields and entry type (and every other setting / config) as YAML files in a config folder. Those can be version controlled, merged and reasoned about easily.

It seems like this should be doable via a module in PW. I have a work-in-progress module that does something like this. It's not complete and haven't looked at it a while but it seemed like a promising avenue at the time. I think if the concept could be illustrated in a module then Ryan might see the benefits and then hopefully incorporate the features into the core.

Well yes, that's also very easy to achieve with RockMigrations.

[MarkE]

Thanks for all that @bernhard. I don't want to perpetuate a to-and-fro and would really welcome input from others too. I do want to say that I, for one, appreciate what you have done, even if I decided to go a different route.

I will take another look in depth at RockMigrations, as it has clearly developed quite a bit since I first looked at it. However, the module info says that it has been archived and that you are working on a new version. Is it better if I wait for that? Also:

• Is there any documentation of best practices beyond the examples? For example, you mentioned putting migrations inside classes and, in particular, custom page classes, which sounds like a great idea, where it makes sense to do so.
• I use PhpStorm, which has many of the same features as your IDE. The snippets sound especially useful (Live Templates in PhpStorm). Are they available anywhere? (I note that there is what appears to be an old GitHub repository for some PW snippets, but I'm not sure anyone ported those to PhpStorm either).

Maybe you can reflect on some of my comments about achieving broader appeal? I well understand that RockMigrations was developed for personal use and that there may be little motivation in meeting other needs (although your contributions to the PW community are already extensive...). Likewise my motivation was to develop something that met my needs. My reason for being concerned about the wider appeal and applicability of a migrations module is to sustain PW as a viable, well-supported and favoured solution, which is ultimately to the benefit of all its users, including me. 

[szabesz]

20 hours ago, MarkE said:

My reason for being concerned about the wider appeal and applicability of a migrations module is to sustain PW as a viable, well-supported and favoured solution, which is ultimately to the benefit of all its users, including me. 

It might be a good idea to somehow persuade more PW devs to give more thumbs-ups this request:

https://github.com/processwire/processwire-requests/issues/230

Anyone reading this and supporting the request, if you have not yet clicked the thumbs-up icon, please do it now :)

 

20 hours ago, MarkE said:

I do want to say that I, for one, appreciate what you have done,

Me too, of course. However, the most important reason why I have not yet looked at Bernhard's modules in details is that he keeps rewriting them every now and then (which is quite understandable but still...) and I certainly have no time to keep up with all the changes. Most importantly, I stick to ProcessWire and to some other (pro/non-pro) modules because I rarely have to deal with any breaking changes throughout the years. That is why I am dreaming of Ryan picking up this request one day in the future.

Edited April 5, 2022 by szabesz
URL fixed

[szabesz]

Sorry, I fixed the URL above which was a wrong one. This is the one I was referring to: https://github.com/processwire/processwire-requests/issues/230

[MarkE]

2 hours ago, szabesz said:

It might be a good idea to somehow persuade more PW devs to give more thumbs-ups this request:

Have done that and commented.

[bernhard]

On 4/4/2022 at 8:26 PM, MarkE said:

Is it better if I wait for that?

Definitely yes ? 

On 4/4/2022 at 8:26 PM, MarkE said:

Is there any documentation of best practices beyond the examples? For example, you mentioned putting migrations inside classes and, in particular, custom page classes, which sounds like a great idea, where it makes sense to do so.

At the moment no, but I plan to create a video that shows the most important parts.

On 4/4/2022 at 8:26 PM, MarkE said:

I use PhpStorm, which has many of the same features as your IDE. The snippets sound especially useful (Live Templates in PhpStorm). Are they available anywhere? (I note that there is what appears to be an old GitHub repository for some PW snippets, but I'm not sure anyone ported those to PhpStorm either).

The ProcessWire snippets are available here: https://marketplace.visualstudio.com/items?itemName=baumrock.pwsnippets (also on github of course), but the RockMigration snippets are at the moment only locally available until I've used them a little more.

On 4/5/2022 at 8:45 AM, szabesz said:

However, the most important reason why I have not yet looked at Bernard's modules in details is that he keeps rewriting them every now and then (which is quite understandable but still...)

Thx for that feedback. That's absolutely understandable and a valid point ? 

Update 2023: I've improved a lot here and try my best to not introduce any breaking changes. My modules follow semantic versioning, so a major version bump indicates a breaking change so that everybody is aware and can check if that is a problem in his/her specific situation. Also there is the rock-monthly newsletter now to communicate the most important news.