Weekly update (Nov 23, 2018)

[ryan]

This week is the Thanksgiving holidays here in the US and it’s one of those weeks where there’s no school for the kids, so it’s a little hard to get work done. I don’t have any major core updates to report this week, so I’m not going to bump the version number up today. However, look for a new dev branch version next week. We will also release a new master version before the end of the year… sometime within the next month. 

Before releasing the new master version, I’m primarily interested in resolving any issues present in the current dev branch that are not present on the current master branch. Meaning, issues that have arisen due to some recent change only on the dev branch (if there are any). So if you are aware of any issues experienced on the dev version that are not on the master, please let me know. Thanks for your help in testing.

Even though it’s been a vacation week, I’ve been waking up early every morning to work on the new PW website. Lots of continuing progress, and I should have another update on that next week along with a new dev branch version of the core. Thanks for all the feedback from last week’s post. Among other things, I caught that folks don’t like the skyscrapers anymore (as a visual element), so I’ve taken them out and agree it’s better without them. I’ll have some updated screenshots next week. 

Off topic, but was so excited I had to tell someone. I got a computer upgrade this week after 4 or so years of working off the same laptop. A few keys on my laptop keyboard recently stopped working (letters in the word “ProcessWire” — I wore them out), so I’ve been using an external keyboard plugged in. That’s been working alright, but made it hard to see the screen since I can’t sit as close with an external keyboard in front of the laptop. It was getting a little tiresome to work on, the keyboard wasn't repairable without rebuilding the whole laptop (costly), and it was basically time for an upgrade, but computers are expensive and I was resigned to waiting another year. Over these Thanksgiving holidays I found out a family member had bought an iMac a year or so ago and didn’t like it, so they were going back to a PC. I said, “hey why don’t you sell that iMac to me?” We came to an agreement. Now I've got it here and am moving my development environment over to this newer computer, and have been working off it for a couple of days and loving every minute of it. It's going to help out a lot with developing ProcessWire. 
 

[HMCB]

Congrats on the new machine! Anytime I add something great to my workflow, it energizes me.

[MrSnoozles]

Congratulations on the new iMac, I'm happy to hear you love it.

And as always: thanks for the update

[psy]

Love my iMac and happy to know you love yours too @ryan. Biggest hurdle for me going from Windows to Mac was - really? it's that easy????

[Pete]

I  upgraded my Windows desktop earlier in the year for the first time in 6+ years as the motherboard was finally dying and the speed difference was immense.

Now I'm the slowest component in my dev environment by a long shot ?

[Kiwi Chris]

Some people think that anyone who works in any form of IT is made of money, but sadly it's not always the case.

I know the feeling about dying laptops. About a year ago, when it was just over 3 years old, the motherboard on my laptop died. Fortunately I was able to track down a replacement part on Ebay, and after carefully removing a lot of screws, swapping over the CPU and heatsink, and convincing Windows that I wasn't running a pirated copy, I managed to get up and going again.

My desktop is a bit long in the tooth too, but since I upgraded the hard drive in my laptop and added an external screen and keyboard, the desktop tends to get less use now, although it was a lifesaver when I needed to get online to order the replacement motherboard for the laptop, and do some work while I waited for the part to arrive.

[teppo]

Congrats on the new computer, Ryan.

I'm assuming that this is related to moving onto the new dev setup, but it almost calls for one of those "yo dawg, we heard you like ..." meme pics, amirite? ?

[Noel Boss]

Thanks Ryan for the latest update and listening to the voice(s) the community! Congrats to your new iMac! As @HMCB mentioned, it's always nice to work with a tool that you can admire, makes working so much more fun.

@ryan Speaking of nice tools, want to share something I came to love very much, would hate to switch back to the old way of having installed my dev-stack locally: 

I switched to AWS Cloud9 (fully cloud based development stack: IDE, Server, PHP, DB… Amazon…) for basically all of my development and love it, it makes you basically machine (and OS) agnostic. I can continue anywhere in the world with my development with only a decent browser and internet connection. If you've got the time, check it out. For me and working in a team or with collaborators (community) its so convenient!

Some of the amazing thins you can do:

- Rapid prototyping: Instantly clone a project with all settings etc, try something, throw it away

- Share the whole stack with someone, no need for the other person to install anything, simply share a url, everything is there, Server, Shell, IDE, DB…

- Collaborate in realtime on a project (google-doc style)

- Create a template container and quick-start your project

…

And since PW basically does not story any urls in the DB, I'm so quick in duplicating Projects with new URLs etc.

Ps. If you're new to the Mac, usually what people do is; they know a tool from the windows world and if there is a Mac version of said tool, they install this one – but many times there is a better native Mac App that does not even exist for windows and you'd therefore don't know it. Look out for these. As a simple example: Looking for an FTP Client you could easily end up with some crappy windows/java port – or you'd end up with Transmit… It's worth looking for these quality alternatives. Also, for example for PDF (Adobe Acrobat) handling, or image annotation, or screenshots or screen-capture or… etc > the Mac can already handle it for 95% of the required work with native Tools like the Preview-App or system-wide features…

[HMCB]

7 hours ago, Noel Boss said:

I switched to AWS Cloud9 (fully cloud based development stack: IDE, Server, PHP, DB… Amazon…) for basically all of my development and love it, it makes you basically machine (and OS) agnostic. I can continue anywhere in the world with my development with only a decent browser and internet connection. If you've got the time, check it out. For me and working in a team or with collaborators (community) its so convenient!

@Noel Boss interesting. And you use Amazon for your live PW sites too? Does Cloud9 let you go from development to EC2 (?) so you never leave the Amazon ecosystem?

And I can vouch for Transmit. I’ve been using it for over 10 years.

[Noel Boss]

Hi @HMCB – no, we are still on the "old" C9 before it got acuired by Amazon – but now you can do so, never leave the Amazon ecosystem. Currently we push to git and pull on the live server – all directly from the c9 instance via Shell. We use some little scripts together with wireshell to pull data from our live system back to dev.

What we still very much miss is a good (native) way to stage and release DB / Shema / Field / Page updates / Templates. Would be interesting how @ryan does it. Thats one of the only drawbacks to PW, that crucial parts of the system are stored in the db. There is a module that stores fields and template settings in json files – but it does not work with repeater matrix…

[arjen]

Hi @Noel Boss have you looked at the migrations module? I've used this on a few projects and it works really great. I usually install this module when most of the templates/fields stuff is ready for a first launch so you don't have to enter all fields manually at first. 

Correct me if I'm wrong but it should work with every module since it uses PW's API. Also the cli is pretty sweet too. 

[LostKobrakai]

3 hours ago, arjen said:

… so you don't have to enter all fields manually at first.

After a while it became even easier for me to just copy&paste together new fields/templates instead of creating them in the UI. There's certainly a transition phase, though. The UI I use mostly for when I'm not sure how to build something yet. For any project with a somewhat longer period of active development/maintainence I'd really suggest anyone trying it.

[Noel Boss]

Yep, I saw that, but its way more (and somehow less) than I need. I just wish there would be an easy way to copy my current setup to a live system without me needing to code stuff and adding more steps to my workflow… auto export would do this but does not cope with pro fields as of now – but I might just go about trying to fix it one day ?

I appreciate the work of you @LostKobrakai and can see how one can get used to this workflow, but for me it's overkill – and underkill at the same time ?  I know its a complex topic… I wish to find a middle ground.

[LostKobrakai]

The problem is that a middleground is even more complex. Either your prod environment doesn't change (in terms of what's in the db) then you can simply use db dumps, or your production environement's db does change concurrently to the dev db. The same is the case if you're working in a team of people as well.

If the latter does happen you need some way to know what to update and what not to touch. Migrations is doing that by letting you manually define migrations, which are the stuff, which needs to be changed. This gives you full control and you can basically update anything in processwire by using the api. You can also still do stuff on your dev environment, which is not somehow automatically applied as change to prod. Bonus point: replayablility / reusability.

What you're looking for is some way to basically let the computer do what you've to do manually with my module: detect what's to change. This could work for simple changes like in fields or templates, but does easily get quite complex is the not so simple cases, e.g. repeater matrix. For pages it's already getting not that simple. You might create pages, which are just for your dev work or pages, which need to be created in prod as well. One could manage whitelists or blacklists (of branches), but that's also a thing to maintain. Also if such a module is not super flexible you'll be bound to execute what it thinks did change. Most often such algorythms don't like a human to adjust their results, especially if steps depend on each other.

[Jonathan Lahijani]

I put this feature request in a couple months ago which relates to this conversation about migrating structural changes:
https://github.com/processwire/processwire-requests/issues/230

[arjen]

18 hours ago, LostKobrakai said:

This gives you full control and you can basically update anything in processwire by using the api.

This is the strongest pro to me. No need to rely on other stuff. If a field is created correctly you can create it with the API.

19 hours ago, Noel Boss said:

I wish to find a middle ground.

I guess the middle ground would be to simply be able to also store the config as json files like Advanced Custom Fields does in WordPress. WordPress reads the json fields first and than the db. You can synchronize the fields between other developers. This sounds very similar like Project Config mentioned by @Jonathan Lahijani. A small disadvantage is other developers absolutely need to sync the fields first otherwise things can get very messy. One big disadvantage is  whenever you need to do additional stuff after the syncing. That is why I like the Migrations module. You can easily create other pages, install modules and such. Rollback is pretty easy and everyone can see the changes in the config through the code.

I thought it was overkill at first too. But it may very well depend on the type of projects and workflow.

I am very curious how other developers handle these issues. One click migration is very nice for marketing, but it can never be that easy.

14 hours ago, Jonathan Lahijani said:

I put this feature request in a couple months ago which relates to this conversation about migrating structural changes:
https://github.com/processwire/processwire-requests/issues/230

What are your experiences with Craft and this Project Config?

[bernhard]

1 hour ago, arjen said:

One click migration is very nice for marketing, but it can never be that easy.

Agree.

1 hour ago, arjen said:

I thought it was overkill at first too. But it may very well depend on the type of projects and workflow.

Same here. But using a JSON-like approach can have severe disadvantages. You see that instantly when you try to export/import a set of templates and fields using PW's built in features. You'll likely end up with lots of warnings because PW does not know the correct order of adding all this stuff. Then it can easily happen that it tries to add a field to a template that is not yet created. You don't have this problem using migrations and actually it's not that hard, see the great blog post here: https://processwire.com/blog/posts/introduction-migrations-module/

@LostKobrakai what do you think of integrating migrations into tracy? I've created a field code section that takes the options of a field and turns it into code. I think it should be possible to take a similar approach and create migration files on the fly. So we could create a field, click "create field migration" and have a migration file with all the settings of the created field. This could especially be helpful to new users that are not familiar with all the options they get via the API.

https://processwire.com/talk/topic/15959-image-field-creation-via-api-requiring-field-to-be-saved-before-use/?do=findComment&comment=173706

[LostKobrakai]

@bernhard I've not much time to work on my OSS stuff at the moment, as I'm no longer actively using it in my everyday work. But I'd happy to collaborate if someone wants to make a companion module and make sure the needed functionality is callable on the module.

[apeisa]

Craft will tackle this issue in their next version (see "Project Config"): https://craftcms.com/blog/craft-3-1-dev-preview-is-here

[Jonathan Lahijani]

8 hours ago, arjen said:

What are your experiences with Craft and this Project Config?

Never used Craft professionally... just tried the demo a couple years ago.

[LostKobrakai]

The problem I have with static json or yaml setup files is that it's not migration. It's just a static set of fields/templates/… without knowledge of what came before or after. I have almost always user generated data associated to a certain setup. So when changing the setup there's often the need for something to happen with said data as well, like when adding a new intro field for blog posts one might want to put the first paragraph of existing posts from the body into the intro. Craft seems to see their addition just as replacement of sharing db dumps and it's essentially that. If environments never concurrently change in terms of content, but just structure then some version controlled file for the structure is better than a db dump, which contains both structure and content. But as soon as the content is also affected by changes it's no longer a solution.

[Mikie]

Since I posted a feature request for something similar without knowing this discussion was happening, just thought I'd chime in.

Obvjously json sync has drawbacks, but here is the workflow I have experienced using ACF with Wordpress that works really well:

• LOCAL (clone of REMOTE) | REMOTE (source of truth)
• LOCAL - work on fields and perhaps content / other config such as templates
• LOCAL - check auto generated field json into version control for other devs to pull, or push to REMOTE as part of deployment
• REMOTE - sync field updates in admin, and if other config / content mismatches then things fail gracefully

To explain the last step, say you have added the field to a template on LOCAL that doesn't exist yet on REMOTE, then on REMOTE nothing goes wrong and if you re-save the field on REMOTE it just updates to the current state there. This is about syncing fields, not other config, so things should just fall back to default if relying on other config that could conflict.

I can definitely see how content (not other config) can quickly become a problem when moving outside of this workflow, but if all you are doing is testing things locally, pushing to live, then syncing the database from live back down to local, this is a really quick and efficient way of iterating field updates and sharing with other devs. 

 

 

 

[Mikie]

13 hours ago, arjen said:

A small disadvantage is other developers absolutely need to sync the fields first otherwise things can get very messy.

@arjen I agree with this, it is by no means perfect. Eg with ACF if you have latest database but old json then you will be prompted to sync changes (which would be reversion). I feel like this is a shortcoming with the ACF implementation though, surely field updates could be timestamped? Maybe I am not thinking the complexities through.

However for a single dev its pretty simple, and with multiple devs there should always be a repo and database that can be considered the source of truth. If things get out whack just pull from latest.

[bernhard]

I strongly support @LostKobrakai 's approach. JSON looks nice at first sight, but we already have a powerful API and we MUST use it. JSON is basically only another syntax. It's some kind of an "API" but a lot less powerful. It's also not easier IMHO, because using the API we have intellisense in our IDE. You don't get that when using JSON.

Updates are a pain using JSON. It might be easy to setup new fields and templates, but what if you want to set a new field width in template context? How would that look like in JSON? It's as easy as that with migrations:

$this->editInTemplateContext('basic-page', 'headline', function(Field $f) {
	$f->columnWidth = '50';
});

The example is also in the screencast.

Another one: How would you create pages in JSON? We know the problem because we already have this tool built into PW. You'll always end up with collision warnings needing some more user input (add this field after that, do you want to overwrite or rename etc etc). You can handle all that with migrations.

It is also quite easy to find the correct property names now as we have Tracy (see end of screencast). And it's also easy with intellisense in your IDE.

 

Where I see potential for the future is that migrations could be used in other modules as well. I think it would make sense to move the migrations feature into the core to have a solid base that is consistant, bullet-proof and maintained together with PW. I think it would make sense to have migrations inside modules, not only at one place. We could then build modules, ship some migrations with it (eg for creating pages and fields) and run / revert those migrations from within the module.

I think that solution would be way better than the JSON config approach and I'd love to see this as a huge update to PW in 2019 @ryan ? 

 

PS: Also take a look at the great docs and examples here: https://lostkobrakai.github.io/Migrations/examples/

[szabesz]

26 minutes ago, bernhard said:

I think it would make sense to move the migrations feature into the core to have a solid base that is consistent, bullet-proof and maintained together with PW.

+1