PW 3.0.24 & 2.8.24

[ryan]

This week updates were focused on covering GitHub issue reports and feature requests, plus some great new performance improvements to our page traversal methods…

https://processwire.com/blog/posts/pw-3.0.24/

[pwired]

Quote

10000 pages now takes about 100 milliseconds (0.1 seconds) on the same slow notebook computer.

Thats very Impressive !! Now we have already two good references for answering returning question about Processwire scalability/performance.

[Robin S]

Echoing a comment from Matjazp on the blog: it would be great to hear how these traversal problems were solved.

Maybe we could have a blog post some time that takes an advanced programming challenge like this, or the findMany() method introduced in 3.0.19, and walks through the process of how it was approached. The initial problem/challenge, the hypothesis of how it could be solved, any missteps or changes in direction that happened along the way, and the resolution.

It would be super-interesting to learn how you tackle these things!

[ryan]

@Robin S Have a look in the PageTraversal class (/wire/core/PageTraversal.php), and the _next() method. It's all in there. The gist of it is that it works roughly the same as before, with one big difference: rather than loading all the sibling pages (Page objects), it loads all the IDs instead (integers), in order. It doesn't always have to load them all either, just enough to determine relative positions. Once its found the needed ID(s), it converts them to Page object(s). It also involved upgrades to the PageFinder class so that it could support startAfterID and stopBeforeID options. 

If you look at the end of the PageTraversal class, you'll see an earlier implementation that I worked with for a couple of days before coming across a couple of situations I just could not get it to work in (and became clear it never could), so ultimately had to abandon it. However, since it relies on how sibling pages are sorted (rather than relative positions), it doesn't have to load near as many, making it potentially faster (and more memory efficient) if there are thousands or millions of siblings. Though in my own testing, it was always slower until the scale got really large. Not to mention, it's a whole lot more complex, and can't cover all situations. But I'm leaving it commented out in there for a little while (for reference) since so much work went into it before finding a much simpler solution.

[LMD]

Hi, my first post here on this forum -- I've been experimenting with PW for about a month now and just launched my first (personal) site using the system.  Because it's a site for myself, I've been using the 3.x version of PW for the 'bleeding edge' stuff and this new update is in fact very relevant right now. 

The site is a webcomic, which has 466 pages/episodes in sequence (a number that could potentially grow to the thousands), each of which have a "first/previous/next/last" navigation bar.  Currently, because of the warnings on methods like next() becoming slow with many pages, I fetch the pages for those navigation links using the following code:

// Note: 'comic_epnum' = field containing the episode number (the most reliable field for sequencing)

// Previous episode
$nav_prev = $page->siblings->findOne("comic_epnum=" . ($page->comic_epnum - 1) . ", sort=comic_epnum");

// Next episode
$nav_next = $page->siblings->findOne("comic_epnum=" . ($page->comic_epnum + 1) . ", sort=comic_epnum");

As you can see, to remove the need to load all pages into memory, the 'previous' and 'next' are obtained with the findOne() method on siblings() by calculating which episode number to fetch.

Am I right in thinking that I could now safely replace that code with the following without any potential performance hit?

$page->prev('sort=comic_epnum'); // Previous episode
$page->next('sort=comic_epnum'); // Next episode

We can not use findOne() to obtain the 'first' and 'last' episodes because we do not know what episode numbers they will be (well, the first will most likely be #1, but maybe you want to start on a different number). Therefore, results are obtained from siblings() with first()/last(), however these could also suffer from inefficiency (according to footnote on this page of the docs: https://processwire.com/api/variables/page/ ), so I got around that by using a selector to limit the number of returned pages to one (1) and just reversing the sort to get the last episode.

// First episode (most like #1) - could use $pages->findOne('comic_epnum=1')
$nav_first = $page->siblings("sort=comic_epnum, limit=1")->first();

// Last (most recently published) episode
$nav_last = $page->siblings("sort=-comic_epnum, limit=1")->last();

The update doesn't actually say anything about first() and last(), so I am assuming they haven't been modified.  But, I wonder if that could be something to look at in the future, so that the following could be super efficient too? 

$page->siblings('sort=comic_epnum')->first(); // First episode
$page->siblings('sort=comic_epnum')->last(); // Last episode

// Or even...

$page->first('sort=comic_epnum'); // First episode
$page->last('sort=comic_epnum'); // Last episode

In fact, I don't understand why first and last are an issue with many pages, because they do not need to know the current pages position relative to its siblings.

Anyway, sorry for my long-winded intro to this forum.

In the short while I have been using ProcessWire, I have been very impressed. I'm coming from a background of WordPress wrangling, where the first thing I had to do on any project was remove everything I didn't want (that could be removed).

[Edit: oops, and the website is: http://www.tranquilitybasecomic.co.uk ]

Edited July 5, 2016 by LMD
Forgot website where the code I'm using is active

[LostKobrakai]

If you're calculating the episode number manually anyways, why then use prev/next in the first place? I mean sorting and siblings and all that becomes basically irrelevant if you've a fixed order by epnum. Same for first/last – no need to use anything related to the siblings functionality.

$nav_prev = $pages->findOne("parent=$page->parent, comic_epnum=" . ($page->comic_epnum - 1));
$nav_next = $pages->findOne("parent=$page->parent, comic_epnum=" . ($page->comic_epnum + 1));

$nav_first = $pages->findOne("parent=$page->parent, sort=comic_epnum");
// Get first in reverse order
$nav_last = $pages->findOne("parent=$page->parent, sort=-comic_epnum");

 

[LMD]

Oh! You are quite right.  Thank you.  I got completely stuck into a single way of thinking (that siblings would result in a less exhaustive search) and totally blanked there. I forgot that I could use 'parent=$page->parent', even though I've used that formulation elsewhere for other things -- you know what happens when it's just you, four walls, a computer and nobody to bounce ideas off.

Although, I'm not actually calculating the episode numbers for any purpose other than the navigation so using the new next/prev would make for less code there, but on the hand it might still be more efficient to stick with the findOne method.