Thousands of pages - One site with multiple pages or One install with multiple sites?

[einsteinsboi]

I'm getting to know Processwire relatively well, and now I have a nice big challenge ahead of me and I'm trying to lay out my plan of attack.  One of my peers reckons I should use WordPress Multisite, which I know is definitely a viable option, but I would much prefer to use Processwire.  I know it's not a question of "Is it possible?" It definitely is. It's just a question of "What's the better approach?"

Here's the situation:  I'm setting up a service that will build site pages for multiple clients (aka baby sites) but all under one domain (aka mothership).  The baby sites will use folders, not subdomains. Most of the baby sites will be one page only, but some may stretch to three pages *gasp*. And of course each baby site will have a completely unique look - so there'll be as many templates as there are babies and potentially more templates than babies.  The mothership will be its own site with its own pages discussing how to become a baby and the benefits of being a baby.

So it will look something like this:

mothership dot com - with about 4 of its own pages

mothership dot com/baby1 

mothership dot com/baby2

and so on up to about baby1000 (potentially)

Now here's my thought process. 

NON NEGOTIABLE - Use Processwire

NEGOTIABLE:

1. One PW site, each baby is just its own page with child pages where necessary

2. One PW core install, each baby is a separate PW site running from the same core wire.

PROS and CONS of the two alternatives as I see them:

Alternative 1:

PROS - Really easy to install and just create the babies as pages on the main site tree. Easy to update core and modules/plugins

CONS - Will it become difficult to maintain as the babies approach the thousands? Will it scale or will it crash?  Will it become a future model of the dangers of putting all your eggs (or babies) in one basket?

Alternative 2:

PROS - Would be nice and easy to maintain from one core, since I only have to update just the one wire folder.  Might scale easier since they're different sites, each with just a few pages.  Any other advantages?

CONS - Don't even know if this is doable?

- Would I need to update each site's modules separately? Annoying

Your thoughts and recommendations are much appreciated by the mothership and the many babies, and by me of course

[pwired]

and so on up to about baby1000 (potentially)

Congrats with getting that project.

PW has inbuild pagination for its cms pages (usable as folders, containers, settings, etc) and is made

from the core up to be both fast and scalable. No worries with pw when your site is growing to a couple

of thousand pages. One of the main reasons why many modx'ers switched to pw.

(remember the months of discussions on the forum).

So you can safely go for option 1.

And I am sure the pro's here are going to come up with some extra tips and ways.

[einsteinsboi]

So you think it will scale fine with thousands of pages? Does pagination stretch that far?

[arjen]

There seems to be a hardcoded limit of 999 paginated pages. So make sure you won't surpass that

[pwired]

There seems to be a hardcoded limit of 999 paginated pages. So make sure you won't surpass that

What ? Ryan or any other pro who knows about this, please confirm.

I can hardly believe this because I remember old posts where the exact same questions are asked and pw should be able to function well until a couple of thousand pages. It is time to publish one day the hardcoded limits and performance range about processwire.

[kongondo]

PWired...I think you misunderstand...This is not about limits to how many pages PW can handle...this is about pagination. Big difference 

It's right there in the post Arjen linked to...scroll down to Teppo's post...or see the code below:

https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/modules/Process/ProcessPageView.module#L32

https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/modules/Process/ProcessPageView.module#L264

Btw, Wanze has built a PW site with about 100K+ pages...

Edited for clarity...

Edited August 27, 2013 by kongondo

[DaveP]

So you think it will scale fine with thousands of pages? Does pagination stretch that far?

Would you want pagination to stretch that far? Doesn't sound great for usability to have a user click <Next> hundreds of times...

Or am I missing something? 

[nik]

Yes, there is a hardcoded max of 999 for page numbering (defined here). Built-in pager does give you pagination links further than that, but they only work for numbers less than a thousand. You just have to take care of the rare occasions when for example a search would give more than (999 * pages_per_result_page) results. As the guys are saying in that other thread (I'm sloow today).

But that's just how far the page numbering will go and has nothing to do with how many pages ProcessWire is capable of handling without any problem. We've got a site with 30000 pages and another one with 15000 pages. The bigger one is running nicely even without any kind of caching at the moment - and there is a bit more complex than just trivial search involved there. And no performance problems with the smaller one either, just to be clear =). There are even bigger PW sites around I'm sure. I think at least Antti has one in his hands and most probably others do as well.

Sure one has to think a bit how to approach things to keep everything nice and fast. But it doesn't require any magic to stay on that road. Common sense and some testing along the way to see little things aren't piling up and you should be fine. In my opinion anyway.

And I'm talking about the amount of pages here, not making any statement on whether to have several small sites in one install or not.

[kongondo]

Mary,

I suppose you've read this in the docs? http://processwire.com/api/modules/multi-site-support/

Your Alternative 2: would mean potentially 1000 databases... ? *shudder* Unless you mean multiple sites from the same database? 

[einsteinsboi]

Would you want pagination to stretch that far? Doesn't sound great for usability to have a user click <Next> hundreds of times...

Or am I missing something? 

There won't be any search functionality as such.  The pagination I refer to is in the admin backend side of things.

But that's just how far the page numbering will go and has nothing to do with how many pages ProcessWire is capable of handling without any problem. We've got a site with 30000 pages and another one with 15000 pages. The bigger one is running nicely even without any kind of caching at the moment - and there is a bit more complex than just trivial search involved there. And no performance problems with the smaller one either, just to be clear =). There are even bigger PW sites around I'm sure. I think at least Antti has one in his hands and most probably others do as well.

Sure one has to think a bit how to approach things to keep everything nice and fast. But it doesn't require any magic to stay on that road. Common sense and some testing along the way to see little things aren't piling up and you should be fine. In my opinion anyway.

And I'm talking about the amount of pages here, not making any statement on whether to have several small sites in one install or not.

Thanks Nik. That's very reassuring! What kind of hosting do you have these websites on?

Mary,

I suppose you've read this in the docs? http://processwire.com/api/modules/multi-site-support/

Your Alternative 2: would mean potentially 1000 databases... ? *shudder* Unless you mean multiple sites from the same database? 

Thanks Kongondo. Yes I've read that.  Definitely don't want multiple databases... that would be hell!! LOL. The whole point of this is to make maintenance as easy as possible, which is why I'm leaning towards option 1.  Just wanted to know how scaleable it would be.  Sounds though that PW can handle a site with tens of thousands of pages with no issue  

My next priority is to learn more about the PW user system since eventually this project will have a few admins and I'd like each admin to only be able to manage a particular set of pages and not be able to edit any of the others.  Off to the docs I go. 

Thanks everyone for your replies  Any more thoughts are of course more than welcome

[Craig]

There won't be any search functionality as such.  The pagination I refer to is in the admin backend side of things.

Don't forget that the admin side has a search feature a swell, so administrators can use that to find the exact baby they want. Another option might be to create the alphabet as part of the page tree, and add babies as children of the correct letter.

Example: mothership/babies/x/xavier

If that structure isn't desirable for the actual front-end URLs, you don't necessarily have to use them - you could also have the home template accept "URL segments", and load the required baby from that using the API.

[apeisa]

I think we are in about 100 000 pages with one site. Mostly because it uses discussions module where each post is a page and I imported all the old discussions there on site migration. Everything runs smoothly, running on company VPS with about 20 other PW sites. We had one performance problem with search that was limited to one rootParent branch, but it was quickly tweaked by Ryan.

So no worries with "hundreds of thousands". Anyone has millions yet? How about test site if not real stuff yet?

[nik]

Thanks Nik. That's very reassuring! What kind of hosting do you have these websites on?

Well, they're also running on a VPS with about 20 other PW sites . But no, it's not the same VPS Antti is talking about. The company actually was the same a year ago, but not anymore.

[DaveP]

There won't be any search functionality as such.  The pagination I refer to is in the admin backend side of things.

My bad - I was only thinking front end.

On the one PW site I maintain that has thousands of pages, the backend search comes into play - knowing the structure well, I can hop to any page quickly that way.

[einsteinsboi]

Don't forget that the admin side has a search feature a swell, so administrators can use that to find the exact baby they want. Another option might be to create the alphabet as part of the page tree, and add babies as children of the correct letter.

Example: mothership/babies/x/xavier

If that structure isn't desirable for the actual front-end URLs, you don't necessarily have to use them - you could also have the home template accept "URL segments", and load the required baby from that using the API.

Thanks Craig.  I assume the admin side pagination supports lots of pages since quite a few people seem to have PW sites with huge numbers of pages.  

I will probably just want the url to be mothership/babyname, so shouldn't be too much of a problem I imagine

I think we are in about 100 000 pages with one site. Mostly because it uses discussions module where each post is a page and I imported all the old discussions there on site migration. Everything runs smoothly, running on company VPS with about 20 other PW sites. We had one performance problem with search that was limited to one rootParent branch, but it was quickly tweaked by Ryan.

So no worries with "hundreds of thousands". Anyone has millions yet? How about test site if not real stuff yet?

WOW!! 100 000 pages!! That takes all my sorrows away

[apeisa]

WOW!! 100 000 pages!! That takes all my sorrows away

Just checked: 119 870 is the magic number today

[einsteinsboi]

OK, so I'm satisfied with the thousands of page being a non-issue for PW.  Now I'm having some angst about the templates end of things. Each of these baby sites will have its own template, so that means thousands upon thousands of templates... will PW be able to scale fine to cater for this? Any one experienced this? Any issues?

thanks guys

mary

[kongondo]

Mary,

I suppose the questions are:

1. Why do you need thousands of templates? Are the baby sites structurally different (e.g. one has title and headline field, the other only has title, or different markup, etc) and/or are they aesthetically different (CSS)?

2. Will the baby sites each have more than 1 template? Max number of templates?

In the end, you do not need each to have unique templates. You can have a base template and pull in .inc or .tpl files via the API to  populate the unique portions of the baby sites in the frontend. Soma proposed a wonderful "template delegation" approach that would fit your project nicely, I think. Have a read here. http://processwire.com/talk/topic/740-a-different-way-of-using-templates-delegate-approach/. There's other approaches there as well. So, instead of ending up with thousands of templates, you could end up with thousands of template file includes. Even then, I should think you can still cut down on the numbers depending on the uniqueness of the baby sites. It just needs careful planning/mulling over.

Having said that, I don't think having thousands of templates will have any performance issues in the backend. I could be wrong on this one since I have never tried it though . You'd probably have to organise them in such a way you could easily edit them. You could use tags to group them in the backend. You could even go fancy and create a simple module that would list your templates in a table, maybe with pagination + search function to filter the list, + a click to edit a template (add fields, etc). The link could take you to the PW template edit screen or could open a modal to edit the template. This would be sort of extending the PW template backend view. We can help you get there if you want to take this route. 

Just my 2cts, there will be other thoughts am sure...

[ryan]

Having said that, I don't think having thousands of templates will have any performance issues in the backend. I could be wrong on this one since I have never tried it though 

Technically I suppose you could have thousands of templates, but that's not where the scalability focus is. PW is designed to scale to unlimited pages, but not templates or fields. There's also the development factor–I don't think one could effectively develop a site with thousands of templates because that's just too much to keep track of. Consider the computer dictionary definition of Template: "a preset format for a document or file, used so that the format does not have to be recreated each time it is used." That describes templates in ProcessWire–something common so that you don't need huge quantities of them.

Mary–do you think you need thousands of templates for layout reasons, field reasons, or access control reasons (or something else)? I think you'll be able to achieve what you need while only having a few templates, but want to better understand the foundation of the need. 

[einsteinsboi]

Mary,

I suppose the questions are:

1. Why do you need thousands of templates? Are the baby sites structurally different (e.g. one has title and headline field, the other only has title, or different markup, etc) and/or are they aesthetically different (CSS)?

2. Will the baby sites each have more than 1 template? Max number of templates?

In the end, you do not need each to have unique templates. You can have a base template and pull in .inc or .tpl files via the API to  populate the unique portions of the baby sites in the frontend. Soma proposed a wonderful "template delegation" approach that would fit your project nicely, I think. Have a read here. http://processwire.com/talk/topic/740-a-different-way-of-using-templates-delegate-approach/. There's other approaches there as well. So, instead of ending up with thousands of templates, you could end up with thousands of template file includes. Even then, I should think you can still cut down on the numbers depending on the uniqueness of the baby sites. It just needs careful planning/mulling over.

OK, so I definitely have a lot more to learn about how templates work in Processwire Thanks for that link I'll read through the thread in more detail and see if I can't wrap my brain around it.  Yes, some of the baby sites will be structurally AND aesthetically different.  After further planning I am sure each baby site will be just one page, so shouldn't need more than one template.

Mary–do you think you need thousands of templates for layout reasons, field reasons, or access control reasons (or something else)? I think you'll be able to achieve what you need while only having a few templates, but want to better understand the foundation of the need. 

The main reason that I was thinking that I'll need many templates is because, as I said, each page on this site will be it's own site of sorts, and will have a different layout and styling from other pages.  For lack of a better example, think of each page as a landing page for a different entity with its own layout, colors, etc, or as I explained in my initial post, as a baby site.  Each baby site (page) may have a different pages under it.

Does that make more sense?

[einsteinsboi]

So after spending some time reading through the template approaches thread Kongondo linked to, I am feeling quite confused    and rather overwhelmed and very aware of my lack of knowledge 

Any simple tutorials out there to guide me with my template situation? Or a simple walkthrough of sorts? As I mentioned the sites will be different in layout and style but I can safely say they will all for the most part have most of the same fields, unless the baby site is a VIP baby in which case it may have a couple extra fields. 

Thanks for all the help 

[kongondo]

Mary, I'll see if I can rustle something up. I would stick with Soma's approach for now.

.....As I mentioned the sites will be different in layout and style but I can safely say they will all for the most part have most of the same fields, unless the baby site is a VIP baby in which case it may have a couple extra fields.

I'm still unsure what you mean by differences here. Maybe provide an (visual) example about the differences? What I mean is this, differences in layout can mean either totally different markup or can mean same markup, but with jQuery and CSS wizardry, a different layout is achieved. 

Another thing, at this point in time, do you have an idea how the baby sites will look like and how many fields they will require? I don't think so, but just asking.

In planning such a site, pen and paper and/or a Spread Sheet are my best friends. I'd write down all the fields that will be needed and note those that will be unique or common...But, I am jumping ahead here. 

[apeisa]

template != template file (usually html & CSS)

You need one template, call it "babysite". Then load site specific CSS when needed and in some case probably site specific HTML too. This is very straightforward and we are here to help. on mobile atm, would provide code examples otherwise.

[einsteinsboi]

template != template file (usually html & CSS)

You need one template, call it "babysite". Then load site specific CSS when needed and in some case probably site specific HTML too. This is very straightforward and we are here to help. on mobile atm, would provide code examples otherwise.

 apeisa

It does sound straightforward when you frame it like that (template != template file).  I'm going to play around a bit more with that idea of one template and then loading site specific css and html, and report back. But if you have a minute to do a code sample for me that would be epic

Mary, I'll see if I can rustle something up. I would stick with Soma's approach for now.

I'm still unsure what you mean by differences here. Maybe provide an (visual) example about the differences? What I mean is this, differences in layout can mean either totally different markup or can mean same markup, but with jQuery and CSS wizardry, a different layout is achieved. 

Another thing, at this point in time, do you have an idea how the baby sites will look like and how many fields they will require? I don't think so, but just asking.

In planning such a site, pen and paper and/or a Spread Sheet are my best friends. I'd write down all the fields that will be needed and note those that will be unique or common...But, I am jumping ahead here. 

Thanks for responding again. Yes I do have a pretty good idea what the fields will be but not what they will look like as these will be different html/css files provided for each babysite by the client.  This is what I mean by different. Each babysite has a different html/css template to be provided that is out of my control, I just have a very good picture of what fields will be on each at a minimum

[kongondo]

Mary,

Bear in mind though that 1 template = same fields for each baby site. That will not be your situation, I think, since some sites will have fields that others do not have, no? So, you will have several templates, just not thousands! since there will be commonalities.

What Antti (=Apeisa )  is describing is essentially what Soma illustrated in the thread I linked to

This is a basic example, and there could be better ways of doing this. 

Each baby site structure, i.e. the HTML could be added to a  .inc file that will be included by PHP to your template file, depending on the page being viewed. The .inc file will also have PHP, especialy PW API in order to output the baby site page's fields. In this .inc file, you may wish to only includethose fields that are unique to a baby site. For instance, all baby sites will have a "title" field. No need to incude that in the .inc file. Instead, include that in a "main.php" as Soma described. Key question is, how do you load the correct .inc file for each baby site? Assuming you have a site structure as follows:

Home

About

Services

Baby1

Baby1-Child1

Baby1-G.Child1

Baby1-G.Child2

Baby1-G.Child3

Baby1-G.Child4

Baby1-G.Child5

Baby1-Child2

Baby1-Child3

Baby2

Baby2-Child1

Baby2-Child2

Baby2-Child3

Baby2-Child4

Baby3

Baby3-Child1

Baby3-Child1

Etc..

In order to select any of the above pages in the PW tree, you have several matching criteria on hand. You can select pages based on their parents, IDs, templates, etc. In our case, we want to make this simple. We want a selector that will match each unique Baby site including its children, grandchildren, greatgrandchildren, greatgreatcgrandchildren, etc. That selector can be the $rootParent. This is the "ultimate" parent closest to the "Home" page (i.e. closest to page->id=1). Hence, the rootParent of "Baby1-G.Child5" is "Baby1". The rootParent of "Baby2-Child4" is "Baby2" and the rootParent of "Baby2" is itself, i.e. "Baby2", etc.

Armed with this, using the delegate approach, we can include the correct .inc file for each Baby site by giving them unique names. E.g., the .inc file for Baby3 would be, you guessed it, "Baby3.inc". You may choose to name it using the actual name of the Baby Site, .e.g "baby-3.inc" (note: lowercase and hyphen; i.e. PW page naming). Same, for others, e.g. "baby-1.inc". In your "main.php" you would load the correct .inc file as follows, for example:

include($config->paths->templates . "views/{$page->rootParent->name}.inc");

This assumes that you have a folder called "views" within your /site/templates/ folders. In the case of Baby1 and its descendants, when viewing those pages, PHP would load the file "baby-1.inc". Inside this file would be, for example the inner markup of the site Baby1. Of course, this assumes main.php will have the rest of the markup, e.g. the header and footer stuff. This also does not directly answer the question, what if there are variations in the Baby site's pages themselves? I.e. the "main page" of e.g. Baby1 could be structurally different from Baby1-Child1. This complicates matters but you can deal with such variations in the .inc file itself since each of your Baby sites will not have that many pages, no?

This is a very basic example. It boils down to how you will organise your site. I have not tested this approach for speed/performance but so far, it has worked well for me. There are other alternatives for loading the correct HMTL, as you would have seen in the Soma thread + others not mentioned in that thread. Important to note is this: PW does not prescribe any method over the other. It boils down to your preferences and sensibilties 

As for CSS, this is one way of loading CSS files:

<link rel="stylesheet" href="<?php echo $config->urls->templates?>css/style.css" type="text/css" media="all">

You can easily apply the same logic of the .inc file here. E.g., by having a CSS file called baby-1.css that will be dynamically loaded when viewing Baby1 pages (i.e. replace the "style" with {$page->rootParent->name}.css, for instance.

Hope this helps. Like I said, it is just but one example. Others will have other views but this should give you an idea....

Edited September 20, 2013 by kongondo