Upgrading ProcessWire

[CaelanStewart]

Hi,

To start off, I'm not talking about upgrading from ProcessWire 2.5 to 2.6 or whatever, I'm talking about a dev/staging/production cycle system. That's all well and good, and works with most of our sites, but one database design aspect of ProcessWire is incompatible with this process. Each user is a page, therefore users do not have a dedicated table in the database that belong to users only.

The way I normally see web software designed is that users are "special" and have their own table to be stored in, so when it comes to the time where we put new site features live, we just drop the users table on the local DB and then export the users table on the live DB and import to the local DB.

This is because we do not want to disable registrations on the live site as we perform upgrades to the site.

I know I could iterate through the users on the live DB, put them all in an asssoc array and then save that array to a file with serialize or json_encode or something along those lines and save user profile photos with a file name corresponding to the user's page ID (and other user created pages), and then run a reverse script on the local DB to add those users back in. Would this work with ProcessWire, or is there more to take into consideration, like references to users in different pages that existed before the upgrade process?

We built a site for a large company, each staff member has an account. Each user has profile photos, and each user can submit suggestions for news articles that are saved with PW, those are the existing references I speak of. There is a blog, with several different templates for different types of article, I.E. downloads, events, competitions which all have different types of info (it does not use the PW blog plugin BTW, we looked at it, but it was not suitable). Its essentially an online intranet, so to speak.

There is an accompanying mobile app which interfaces with PW through an API I wrote

We're not talking about a small user base, there are 250 users on here and it is growing as more and more staff register. There is a custom registration system which adds users in the same was as expected, so user's in the context of this site are no different to any other PW site.

What we are going to do in a worst case scenario is put the live site offline for a couple of hours while we download the live DB, repeat all of the changes I made to fields, templates and what not (I logged every changed I made to fields, templates and pages as we foresaw this issue).

The version of ProcessWire we have is 2.5.3.

I must also say, ProcessWire is dealing with the usage quite well, the site is still relatively quick to load.

Thank you in advance for your time, it is greatly appreciated.

[LostKobrakai]

Users aren't actually the only part where this workflow is flawed, while maybe it's the most obvious. You've the same problem with every bit of dynamic page content in contrast to more "static" pages, that are manually created for things like structure. There's no real way to update only the first or only the latter – it's one pagetree full of pages. I don't know any easier way to update a page other than using maintenance downtime to do so.

[pideluxe]

Hi Caelan,

at the moment i think the easiest way would be to disable the site while updating, make changes to fields and templates and upload your local template-folder to your live-folder.

But to really answer your question, we need some more infos from you :

Why do you want to copy the live DB to local DB? Do you need pages from the live DB to be backuped? Especially the users, why do you need to import them in the local DB?

There a some articles in the forum which deal with this issue, the topic is continious integration or something like this.

[adrian]

This issue is what Migrator (https://github.com/adrianbj/ProcessMigrator) aims to solve (although not for users just yet). It works very well in most scenarios - there are just a few situations that still need some tweaking. It works by converting page IDs to named json records, named assets folders etc so that they can be imported into a new site without causing clashed with existing page IDs.

Unfortunately for your purposes of migrating user accounts, there are currently several checks to exclude pages under Admin (I wanted to prevent it messing with this stuff), but I do think they can probably be removed - carefully!

Anyway, not a complete solution for you right now, but something for you to investigate.

I haven't found much time to work on it of late, but I am still intending on finishing it up sometime soonish.

[LostKobrakai]

@adrian

Migrator works for you? I've tried it once, wanted to move a branch of the pagetree to staging, but it did import all the pages, that where linked by pagefields, as siblings of my branch, while all of them where already part of the staging environment. Also I'd like a option to just export pages (trees), without fields / templates, as I like to use the native ex-/import of both.

[adrian]

Migrator works for you?

Of course Seriously though, I find that in general it works really well, but I do need more testing from others to find these situations where it doesn't. Keep in mind that there might be an issue with recent versions of PW (https://processwire.com/talk/topic/8660-migrator/?p=92889) that I need to look into.

The issue you mention might be related to this, however I do also need to work a bit on the behavior for migrating the content page tree for page fields - I need to check to see if that tree already exists before importing so it can update/replace it, rather than adding a new one. 

Interesting that you want to exclude field/template migration - what happens if the fields/templates for the pages you are trying to import don't already exist? I don't think it should be hard to make this an option though.

If you'd like, please PM me your exported json or zip file so I can test - it might help me to discover some other scenario where it isn't working as expected. I really do want to get this stable and released sometime this year.

[LostKobrakai]

With lot's of page tables and page fields (which even get hiccups from time to time with the nativ import) I like to port templates and fields manually. My current project is already in a state where lots of things are manually copied over (lot's of id inconsistencies) so maybe I should give the module another try on a smaller, more controlled environment first.

[CaelanStewart]

Users aren't actually the only part where this workflow is flawed, while maybe it's the most obvious. You've the same problem with every bit of dynamic page content in contrast to more "static" pages, that are manually created for things like structure. There's no real way to update only the first or only the latter – it's one pagetree full of pages. I don't know any easier way to update a page other than using maintenance downtime to do so.

Downtime is simple not acceptable, the site is used by executives of the international corporation this site was made for. There shouldn't be many more big updates like this, but I would just like the next iteration process to be easier.

Hi Caelan,

at the moment i think the easiest way would be to disable the site while updating, make changes to fields and templates and upload your local template-folder to your live-folder.

But to really answer your question, we need some more infos from you :

Why do you want to copy the live DB to local DB? Do you need pages from the live DB to be backuped? Especially the users, why do you need to import them in the local DB?

There a some articles in the forum which deal with this issue, the topic is continious integration or something like this.

Read reply to quote above.

And to answer the DB question, we copied the live DB as we cannot work on the live site and risk causing downtime, the mobile app is tied to the site, and the changes are substantial, so the app is due an update too.

This issue is what Migrator (https://github.com/adrianbj/ProcessMigrator) aims to solve (although not for users just yet). It works very well in most scenarios - there are just a few situations that still need some tweaking. It works by converting page IDs to named json records, named assets folders etc so that they can be imported into a new site without causing clashed with existing page IDs.

Unfortunately for your purposes of migrating user accounts, there are currently several checks to exclude pages under Admin (I wanted to prevent it messing with this stuff), but I do think they can probably be removed - carefully!

Anyway, not a complete solution for you right now, but something for you to investigate.

I haven't found much time to work on it of late, but I am still intending on finishing it up sometime soonish.

Well, user's are the most important thing here. I will have a look at that and see if I can't modify it. Will let you know if I attempt it and get somewhere.

[Soma]

It's almost impossible to do updates to a high frequent live site without downtime. At least not with common setup php mysql. We have a project that has always user on the site browsing, ordering, commenting etc. Every time I even just upload a global template or script/module, there happen to be at least 4-5 errors in that 0,5 sec uploading it. I've been thinking a lot about how to fix that but it's simply not possible. Best is still, even for very small updates, to shut the site for maintenance to make sure there's nothing fancy happening.

[CaelanStewart]

Hi,

We managed to pull this off relatively quickly. I made a log of every field/template/setting change we made to the site locally, I then downloaded the live site and put it offline while I upgraded the live site. I went through the live site and repeated each of the changes in the l had logged down, I then added in the few news article they have and then I was done. It only took a couple of hours, so the downtime was minimal.

It is possible to upgrade ProcessWire, but it can be a bit of a pain in the butt. But in all honesty, ProcessWire probably wasn't the best choice of software for the kind of site I used it for, it was never designed to have a registration system and hundreds of users, it seems like it was designed for only a few users and no registration system. Let alone an API system for use with a mobile app. Nevertheless, ProcessWire has dealt with the strain reasonably well, and with some tweaks and some performance enhancing modules, it runs quite fast with 300+ users on.

[LostKobrakai]

Don't take this as personal offence, but that's not a fair statement to make about ProcessWire. The problems you had with updating your backend are inherit to using relational databases and not specific to the cms. You could've used any of the nosql databases out there, but that's a whole other story and quite different in terms of content management.

Stating that the system was never designed for registrations, a great user base or for mobile api's is a plain wrong accusation. The system was not designed for plug-and-play, that's true, but it gives you the full freedom of implementing registrations and api endpoints just like you need them. At the core ProcessWire is more like a cmf, so what you do with it is up to yourself.

[Martijn Geerts]

it was never designed to have a registration system and hundreds of users, it seems like it was designed for only a few users and no registration system. Let alone an API system for use with a mobile app.

Don't get me wrong but what you state here is plain wrong. ProcessWire is more 'content management system' then most website/blog manage systems. Processwire as Restfull or content deliver API is a great! Handeling 100.000 users is just a breeze. So I would power up my app with ProcessWire just because it's the easiest and best choice ( marketing mode  )

On the other-hand, when developing on this scale, you need time to develop. Customers who give you these jobs should know or will learn that building something 'big' will cost big money, regardless of system.