Selector vs direct database search efficiency

[Hurme]

Hey,

I know there has been lots of discussion on this subject already, but I feel like I need some clarification on the subject, especially in the light of the new $pages->has('selector').

So lets say we have a site with a couple of million pages and we need to find one or more pages from that pile.

1) Usually direct database search would probably be recommended at that point, but does the $pages->has('selector') speed things enough to be considered a serious alternative?

2) If I wish to load a list of all the Names with say one extra field from all the pages, will PW always load the whole page (times million), or is it possible to only get the fields I'm looking for and nothing more?

[Martijn Geerts]

When requesting a page with selector you do not request other fields exept title as it is set to autojoin.

[bernhard]

1 hour ago, Hurme said:

1) Usually direct database search would probably be recommended at that point, but does the $pages->has('selector') speed things enough to be considered a serious alternative?

2) If I wish to load a list of all the Names with say one extra field from all the pages, will PW always load the whole page (times million), or is it possible to only get the fields I'm looking for and nothing more?

Would be nice if you could try RockFinder2 and share your results ? 

 

[dotnetic]

8 hours ago, Martijn Geerts said:

When requesting a page with selector you do not request other fields exept title as it is set to autojoin.

If you iterate through the results of pages and use the page object in that iteration (for example if you want to get the value of a field), then the whole page would be loaded into memory, which is very slow if you have many pages. This could also lead to out of memory errors.
I experienced this, before I found @bernhards solution RockFinder2 which is really great. There are some discussions about the performance here in the forums.

[thetuningspoon]

I'm a little unclear what the question is here. A ProcessWire selector ($pages->find($selector)) gets converted to a "direct database search", so there is no issue selecting from millions of pages, unless you're expecting to get thousands of pages back from it. Can you clarify what you're trying to accomplish?

Edit: Sorry, I should have read your second question more carefully. As others have mentioned, only fields with autojoin turned on would be loaded with the pages (names are always included). After loading you could use WireArray::explode() to get an array with just the names. You might also try Pages::findMany() to prevent memory overload (though that will probably not work with explode) . But if you're talking about thousands or more you'd be off going to RockFinder or straight SQL.

[Hurme]

Hey,
Thanks for the replies all.

@bernhard We'll take a look at Rockfinder 2. We don't have that many pages yet, as this is more about future proofing the site, but eventually it has potential to grow very large indeed.

I'll get back here once we are further along.