Nico Knoll Posted September 29, 2012 Share Posted September 29, 2012 Hey, I once saw a post about some extreme tests with processwire like more than 100,000 pages etc. But I can't find it anymore so my questions are: - how fast is processwire with like 1,000,000 pages - is it possible to split it on two servers (when the traffic should be to huge for one) - ... I'm working on a network about music videos (@renobird knows what I'm talking about ) and it might get really huge... / Nico 1 Link to comment Share on other sites More sharing options...
teppo Posted September 30, 2012 Share Posted September 30, 2012 (edited) Sorry, can't really answer your first question, haven't seen a setup that huge so far. So far large sites have worked pretty nicely (talking about <100,000 pages here) with the exception of repeaters causing trouble here and there. I've heard that there are some really nice speed improvements coming regarding those, though -- but I'd still generally encourage you to avoid them when building this kind of site (unless someone can prove me wrong on this one.. ) Anyway, first of all I'd suggest you to take a good look at ways to maximize site speed without applying any complicated tricks -- cutting HTTP queries to minimum, compressing CSS, JS and images, using cache as effectively as possible etc. Also: if you need to host your music videos yourself (though hosting at YouTube or similar service would generally be a much better solution) I'd create an external server just for static content like that; "videos.example.com" or something like that. All the big guys are doing stuff like this already. Regarding splitting PW site on two (or more) servers, that should be possible in theory, but would require a lot of work and could be a little "hacky." The main problem here would probably be assets; you could have multiple servers running PW while still sharing one database server (and a load balancer in front of those servers) but you'd need to find a way to keep assets synchronized. I don't really have a viable solution for that at the moment, though it's something I've been looking into too -- currently handling stuff like that with rsync, but that's nowhere close to "real-time." I think someone somewhere (?) mentioned that PW sessions can also be saved in database, which would help a bit here - don't know if that feature is still in dev branch though. Another solution would be to run a main site on one server + separate "subsites" on others (example.com, genre.example.com, anothergenre.example.com, users.example.com, auth.example.com, comments.example.com etc.) but that'd make communication, searches etc. between those sites way more complicated and at least require some kind of SSO solution. To be honest quite often it just makes sense to keep things running on one server (with the possibility of separate servers for database + static files as mentioned above) and when requests pile up just put some extra muscle on those servers. That's another reason to love virtualization.. keeps things nice and flexible Edit: the more I think about this, the more I start thinking that the viability of having one PW site run on multiple servers simultaneously is related to how often assets are added / changed / removed. If your typical user doesn't have the right to manage assets, it would be relatively simple to implement a server-side watcher that would sync assets between sites / servers. Rsync (for an example) has an option to only sync files that have been added / changed, thus you wouldn't have the need to sync all the files between servers whenever something is changed. In this setup cache files could cause some trouble, though -- not sure how to handle those properly. A filesystem watcher could be a self-made server-side combination of rsync + cron (non-realtime) or some of the already available applications, such as Lsyncd. Quite possibly you could also have the watcher work as a PW module, hooking into file save operation and then invoking rsync / other sync method via exec() -- not sure if there's an easy place to hook into yet though, just throwing in some random ideas here. Oh, and all of these ideas rely heavily on you having complete control over your servers. I hope that's the case. They're also based on my relatively low knowledge of PW + server administration, so there might be a much better solution available.. Edited September 30, 2012 by teppo 1 Link to comment Share on other sites More sharing options...
ryan Posted September 30, 2012 Share Posted September 30, 2012 ProcessWire should scale very nicely as things grow. You shouldn't see slowdowns from quantity of pages unless you've got a bottleneck in your code. By that I mean primarily: using selectors without appropriate limit statements. Repeaters should also be used carefully like Teppo mentioned. Though the dev branch resolves the major bottleneck there that Nik had found and helped to fix. I don't see any problem with repeating your site over multiple servers so long as you are designating one as the master where actual changes are made. Pete may be able to speak to this better than me, but apparently some web hosts scan scale your server virtually, to quite a large scale that [apparently] spans several servers while behaving as one. But this is something I've not experimented with yet. I think someone somewhere (?) mentioned that PW sessions can also be saved in database, which would help a bit here - don't know if that feature is still in dev branch though. This is actually already in the dev branch. Session handling can now be done by plugin modules, and it comes with a database session handling module. But you can pretty easily implement other types of session handlers using the same interface. 3 Link to comment Share on other sites More sharing options...
Nico Knoll Posted September 30, 2012 Author Share Posted September 30, 2012 Nice. Thank you guys for quick infos Link to comment Share on other sites More sharing options...
Recommended Posts