Multiple site monitoring

[Pete Jones]

We have a few PW sites and plan to increase this number. I'm looking for a multiple site monitoring tool similar to https://managewp.com/ where you can see a snapshot of all of your sites, the versions of the core, modules etc and also a way to update them centrally?

ManageWP also handles malware scanning, database optimisation. 

Is there anything available for PW?

Thanks

Pete

[LostKobrakai]

There isn't, but also there's not much need for it by now. With none known security issues the need for centralized updates is quite slim. Doing updates just for the sake of it doesn't make much sense. 

[Pete Jones]

I'm pretty new to PW but have 15 years Joomla and Wordpress experience and have watched those platforms expand to a point where you need a good tool to centralise maintenance.

Having no known security issues and therefore no reason to apply updates seems a little cavalier. Unless you're looking to keep backwards compatibility I would always think you're best to apply an update when it comes out?

Not knowing PW that well, I could have this wrong though - please don't flame me

[DaveP]

Hi petejones, and welcome.
 
What LostKobrakai said, plus...

• PW really is a different beast to WP (and Joomla, or Mambo as it was when I first got involved with it). PW has had no known security issues, as ProcessWire or in its earlier incarnations. Background here. Unless you want the extra functionality that updates bring, there is no need to update an existing installation.
• Ryan explains one significant contributor here.
• I know this sounds strange, but PW doesn't need maintenance. It just works and carries on working and doesn't become broken for arbitrary reasons. Alright, on shared hosting (for example) there is the possibility that something could happen, but that would be because of poor security practices by the hoster, not PW.

[Pete Jones]

Thanks Old Bloke.

What a refreshing CMS! I'll have a read of those links.

[LostKobrakai]

Why would you want to update a running system and have the slight chance to break something when there's no need to. If you need any of the newer functionality you're needing that for a new feature anyway, so you're already working on a specific website. Same goes for issues with hosting or own security issues one might have introduced. 

Only thing I'd say is really interesting is having a central point to initiate things like database optimizations or other less critical maintenance tasks, but that's really up to the dev / agency to create, because everybody would need different things to run in different circumstances. Running updates is certainly the least needed at the current stage of processwire.

[Pete Jones]

Why would you want to update a running system and have the slight chance to break something when there's no need to. 

I guess that's where PW is different from most CMSes. Most mainstream CMSes need updates to prevent some kids with a load of scripts trying a load of known exploits against your site.

[teppo]

I guess that's where PW is different from most CMSes. Most mainstream CMSes need updates to prevent some kids with a load of scripts trying a load of known exploits against your site.

That's true. Still, there are some reasons why you might want to update sites anyway: 

• If you're hosting and supporting multiple sites, it's more overhead when they have different versions of the system running
• Updates often bring in new features – such as hugely improved image management tools – that your customers might want to use
• Not having known vulnerabilities in the core or third-party modules in the modules directory so far doesn't mean that these can never ever appear; especially when you're working with third party code, it's good to keep track what you're using and where, and for this reason alone a centralized management platform can be quite useful

We're hosting quite a few sites and for this purpose I've built an in-house solution that does pretty much what you've described here. The focus is on making management of a large group of sites effortless and providing automation for common tasks, but this platform is also used to keep track of site-specific features, versions, usage, and so on.

There's little chance of opening our in-house solution to larger community, but the thought of building something similar and open-sourcing it has crossed my mind a few times. Potential user base for this would be somewhat limited for the time being, but this is definitely something to keep in mind in the future

[adrian]

While I agree with earlier posters about not needing to update PW installations, honestly I agree more with teppo - I want older customers to reap the benefits of improved admin functionality. And course we are not just talking about the PW core here - there are also often 3rd party module updates - sure these updates might just add new functionality and there is always the chance that they might break something, but if you do your due diligence in checking the revised versions, it is nice to be able to update across all your sites at once.

I actually had/have some plans to build something like this - but am waiting to see if the demand is going to be there.

Module Toolkit is something I use in the meantime for updating all 3rd party modules on a site at once - it can be a significant time saver. But be aware that this module is currently alpha/early beta - I would like to get back to it and release it properly, but juggling priorities as usual

[tpr]

PW is different because updates are mostly installed by devs who finished their projects much before deadline and would like to have some extra fun time in the admin

[Peter Knight]

I think a central manager (ProcessWatch ?) is a great idea and something I've needed in the past.

My collection of PW sites is growing rapidly and a birds eye view if all of them would be a real bonus as we manage more and more.

Are there recent developments with PW3 which allows it to share data with other PW sites? Maybe the time has never been better to have a central PW install with the:

1. Ability to see which version is running and what latest Dev and Stable version is

2. Ability to upgrade , backup database etc

3. See a modules installed and their version. See if a new version is available

4. See the health of a site (some kind of ping or status - live or non responsive)

5. See who is logged in per site

6. See latest changes per site

I think all of the above are already available within a PW install via native functions and Module so I wonder how they could be harnessed on a central console.

[gurkendoktor]

I would suggest to split this. In my opinion, all the points except 4 can be in the backend. For 4.) there are dedicated monitoring tools which measure exactly this: availability, load, status of services, memory consumption, … you name it. This again can be made available by API. 

Besides, you should _always_ be able to update. ALWAYS. Whether you feel you need it or not. One day there might be a breach no one saw coming and you do want to be able to install the newest version in the least time possible. So in your own interest, always update, always make your modules update-friendly, and always keep your hands of fiddling around in the core.

[BitPoet]

Yes, all these points would mean creating a monster. In terms of database backups or malware scans, those are highly platform specific and not something easily implemented in a module.

A centralized overview actually makes sense to me. Showing.PW and module versions and outputting some statistical information is also quite doable. Displaying a momentary status can be done, but continuous monitoring is not something that you can build on top of PW - for this, you'd need to add some layers under or next to it.

I've started tweaking together a set of modules that could serve as a base. I'll try to add the necessary hooks to allow extending it tomorrow and put an alpha release on github. If nothing else, it's an interesting conceptual challenge

[gurkendoktor]

Actually, specifially DB backup should be easy as you have access to it anyway. All things server related should be covered by stuff like icinga / nagios or 3rd party system like newrelic. The data these services provide, you could however put in your admin area – together with PW related data.

I think there is an easy update module already and afaik it also shows when modules are out of date. But go ahead, better is the enemy of good

[kongondo]

Maybe abstract different tasks to different (sub)modules? Something similar to the tools module by netCarver et al. (can't remember its name atm). Then one could add only the bits they need or use other third party tools for some tasks. As for the monitoring system itself, I would probably install it on a ProcessWire site with no frontend traffic, basically, only  backend whose only work is to monitor, inform and issue commands to the sites being monitored.

Oh, welcome to PW and the forums petejones

Edited January 24, 2016 by kongondo

[BitPoet]

Okay, so I've done it. Two sets of modules, one for the server side, the other for the client side. There are still a lot of ToDos, like adding dedicated endpoints on server and client (I'm currently hooking indiscriminately into ProcessPageView::execute, but I'd like to go the route of only loading the listeners for this template) or adding some kind of challenge-response mechanism to keep nosy third parties out.

The modules are up on github:

Server component

Client component

There's also a screen cap in the readmes.

Both module sets have lots of hooks, so component modules should be easily able to extend the basic functionality.

Currently, the online functions arealdy built-in are the setup of the management connection on both sides and getting basic client information. I haven't really looked at the upgrade or db backup modules, so I've got no idea how complicated remoting them would be. But the sky's the limit. Comments, critiques and pull requests welcome!

[DaveP]

So, 48 hours or so after a question that initially got the standard "PW != WP" answer, there's been a reasoned discussion, people have come round to seeing some merit in the initial premise, and BitPoet has a couple of alpha modules on GitHub.

That is an extraordinary testament to the quality of PW and this community.

[grimezy]

Hmmm... I'm looking for a module that can clean my house and can make me coffee...  

[Pete Jones]

So, 48 hours or so after a question that initially got the standard "PW != WP" answer, there's been a reasoned discussion, people have come round to seeing some merit in the initial premise, and BitPoet has a couple of alpha modules on GitHub.

That is an extraordinary testament to the quality of PW and this community.

I have to agree. As first impressions go, this is excellent. Having spent years at the helm of Joomla sites and having just moved to an agency that has a WP back catalogue and have been dabbling in PW, it looks like they picked the right platform and more importantly community!

[Peter Knight]

Okay, so I've done it. Two sets of modules, one for the server side, the other for the client side. There are still a lot of ToDos, like adding dedicated endpoints on server and client (I'm currently hooking indiscriminately into ProcessPageView::execute, but I'd like to go the route of only loading the listeners for this template) or adding some kind of challenge-response mechanism to keep nosy third parties out.

The modules are up on github:

Server component

Client component

There's also a screen cap in the readmes.

Both module sets have lots of hooks, so component modules should be easily able to extend the basic functionality.

Currently, the online functions arealdy built-in are the setup of the management connection on both sides and getting basic client information. I haven't really looked at the upgrade or db backup modules, so I've got no idea how complicated remoting them would be. But the sky's the limit. Comments, critiques and pull requests welcome!

Really impressed. 

For some reason (before you built this) I assumed the Monitor app would be accessed via a private web page.

Of course, it would naturally be part of the Admin as you've done. Not sure why I jumped to that conclusion :-/

[David Beesley]

Hi folks,

I have been asked by Pete (my boss) to pick up the ball and run with it, as he is quite keen to have a centralised solution for monitoring versions and updates.

Having worked with Wordpress, Perch and Umbraco in my time, I must say this is the nicest CMS I have has the joy of using (Perch is a very close second). It seems to be very well maintained by a core of hard working individuals, but has yet to reach the size where it is a target for crackers and script kiddies. That said, with it being our CMS of choice, we don't want fragmentation to be come a problem. I remember the groans that went around the office of a former employers when we needed to make a change to an Umbraco 4 website.

I'd be interested to know if anyone has had any success with developing a solution for this?

@BitPoet thanks for being so pro-active and working on a prototype, although it looks like you haven't had chance to make a commit to it since January. Understandable, you have to pay the bills and keep the lights on rather than maintaining a side project for some guys you have never met.

Perhaps we could have a quick chat on Slack, DM or similar about the project and see if we can pick it up and help develop a solution with/for you? In the meantime, I'll have a good poke around and see how it works on a test installation.

 

Thanks all

 

[David Beesley]

Hi @BitPoet

Thanks again for you work on these modules.

I have it all installed, the two server components on a Blank install of Processwire, and the two client components on a test version of a current client site.

Both sites are hosted on an internet visible server (on the same IP address).

I have configured both ends, but they aren't talking to each other. When I click "Connect/Refresh Server" on the client website it alerts "Invalid or empty response from server"

 

I have added some screenshots of my configs with sensitive data removed. But the client secret and ping from server have never had any data in them

[BitPoet]

@David Beesley: I haven't tinkered with the modules in a while, but I'll try to find the time to run some tests over the next few days. I'll post back here.

[BitPoet]

It appears that the last commits were quite incomplete, though the newer versions on my dev drive also had a few issues of their own. That's what you get when you replace sleeping with coding once too often

I've updated both client and server module to 0.0.7 now, with partially re-written handshaking logic and a more detailed remote module list feature. I'll be quite busy the rest of the week (including the weekend), but if time allows, I want to tackle two issues in the first half of November: a cryptographic handshake - don't like really plaintext passwords going over the wire - and a documentation and an example module for the remote actions system. The latter will no bring up some more issues that needs fixing/tweaking along the way.

My priority will be on my DatetimeAdvanced and JsonNativeField modules though, as I'll be using these in a production system in the near future, so I wouldn't ask anybody to hold their breath yet.