Max number of pages?

[muzzer]

Hi,

Coming from modx (Evolution) I found one restriction (which was later solved in Revo version) was a page limit of between 2000 and 5000 pages, after which performance apparently took a nosedive. I never fully tested this but it was a restriction I was always aware of and it did prevent be building a couple of sites.

I would be interested to know if there is a theoretical "page limit" in PW, or alternatively has anyone developed a site with 5000+ pages and how well does it perform. How does it affect caching? Thanks

[Soma]

Not sure if there is a limit other than your hardware. Depends what those 5000 pages are doing but I see no problem at all even without cache. It's much faster than MODX. If you take care of what you do with queries in templates and maybe use partial markup cache for heavy queries I think you can do a very performant website within the hundred thousands easily only really limited on performance of the server PW is quite optimized.

Anyway it has never really been tested how far you could go but I heard about sites in the twenty+ thousands and they still seem to be fine. Just recently there happened to be some issue with large scale when saving pages and already got more optimized.

[teppo]

Highest page count I've seen on a production site so far is 30k+ and that site runs just fine. All things considered I wouldn't be too worried -- PW is surprisingly resilient

There's an earlier thread with almost the same question that you really should read; in his reply Ryan explains some of the things you need to consider when building a (very) large site: http://processwire.com/talk/topic/1527-largest-project-performance-multi-server/#entry13779. That performance issue Soma mentioned is discussed further here: http://processwire.com/talk/topic/1508-performance-problem-possibly-a-bug-in-pages-class/ (and yes, related problem / bottleneck has already been fixed.)

[muzzer]

Thanks guys, answers the question very nicely.

[Adam Kiss]

I have an app built on top of PW running, which is currently on 6.5k pages; of those, 3.5k are children of one page (!); The app feels a little slower, but that is probably due to no performance review/check on my side.

Also live generating statistics over those 3.5k pages takes a few (4-8 ) seconds, but it's just because it's always regenerating statistics, iterating the pages multiple times.

[ryan]

I'm running 21k pages on one site right now, and no difference in speed than if it was running a dozen pages. Though of course, I'm not doing anything that loads all those 21k pages at once (I don't think that would even be possible).

[vanderbreye]

is there any limitation?
I'm worried that the PAGE-ID Field is (might be?) a INTEGER and can not be used with >65k pages.. or if there are any other limits in the core...

(Currently I'm running a system with 25k pages, but there are chances to break the 100k limit... therefore i want to be prepared..)
 

[kongondo]

What 100k limit? Somebody here had 500k pages .

1. The page_id field is int(10) - the 10 just means how many digits can be displayed in the db column
2. It is an unsigned integer field so...from the MySQL docs...

All integer types can have an optional (nonstandard) attribute UNSIGNED. Unsigned type can be used to permit only nonnegative numbers in a column or when you need a larger upper numeric range for the column. For example, if an INT column is UNSIGNED, the size of the column's range is the same but its endpoints shift from-2147483648 and 2147483647 up to 0 and 4294967295.

http://dev.mysql.com/doc/refman/5.1/en/numeric-type-attributes.html

http://stackoverflow.com/questions/8892341/what-does-int5-in-mysql-mean

http://dev.mysql.com/doc/refman/5.0/en/numeric-type-overview.html

You will be fine

[DaveP]

FWIW we have an extranet app running on PW, the top 4 parent pages having 2657, 8201, 2850 & 1750 children each. (There is also a non-PW db table that has about 1.6 million rows but that doesn't count.)

There are a couple of templates that take longer to load than I would be satisfied with on a public site, but that is because of amount of processing going on when they load. The admin interface is no different in performance terms than a site with 10 pages total.

There is no caching, as the data are always changing and all users are always logged in.

[MuchDev]

I have a site that has over 20k, I haven't run any benchmarks on it but it still is running like a champ. I'll have around 10k more by the end of the month and I don't see any reason that the site would be much slower. I will on the other hand be adding more resources for the vps so that page fetches are a bit snappier. I suppose you would just want a bit more horsepower for the MySQL back end if your uncached pages are a bit slow. 

[Martijn Geerts]

The bottleneck could be the filesystem when you reach more then 30.000 pages with assets. But ProcessWire has you covered with the pagefileExtendedPaths setting. When you expect your site to grow beyond 30.000 pages with images or/or files enable this setting in /site/config.php

Ryan states that this feature is beta but I confirm it's working without any quicks.

/**
 * Use extended file mapping? Enable this if you expect to have >30000 pages in your site.
 * 
 * Warning: The extended file mapping feature is not yet widely tested, so consider it beta.
 *
 * Set to true in /site/config.php if you want files to live in an extended path mapping system
 * that limits the number of directories per path to under 2000.
 *
 * Use this on large sites living on file systems with hard limits on quantity of directories
 * allowed per directory level. For example, ext2 and its 30000 directory limit.
 *
 * Please note that for existing sites, this applies only for new pages created from this
 * point forward.
 * 
 * @var bool
 *
 */
$config->pagefileExtendedPaths = false;

[MuchDev]

Man that is awesome man, I am going to enable that setting. I foresee my site getting to around 60-70k items as they will most likely be adding inventory in much larger batches now that all they have to do is give me a spreadsheet with the item data and the images. 

[MuchDev]

The bottleneck could be the filesystem when you reach more then 30.000 pages with assets. But ProcessWire has you covered with the pagefileExtendedPaths setting. When you expect your site to grow beyond 30.000 pages with images or/or files enable this setting in /site/config.php

Ryan states that this feature is beta but I confirm it's working without any quicks.

/**

* Use extended file mapping? Enable this if you expect to have >30000 pages in your site.

*

* Warning: The extended file mapping feature is not yet widely tested, so consider it beta.

*

* Set to true in /site/config.php if you want files to live in an extended path mapping system

* that limits the number of directories per path to under 2000.

*

* Use this on large sites living on file systems with hard limits on quantity of directories

* allowed per directory level. For example, ext2 and its 30000 directory limit.

*

* Please note that for existing sites, this applies only for new pages created from this

* point forward.

*

* @var bool

*

*/

$config->pagefileExtendedPaths = false;

Ok, I am not seeing this inside my site/config.php is this something that doesn't exist in the latest versions, or is this something that I should add?

**

I found it in my wire/config.php and added it into my site/config.php. Thanks a ton, now to see if it does anything where I am now .