Rendering of a page list takes too long

[Seb]

Hello ProcessWire community,
as this is my first posting, I want to thank Ryan Cramer and the ProcessWire community for developing such a great software. ProcessWire is the first CMS that fits my needs perfectly and it's a pleasure to work with it. (Also thanks to isellsoap for recommending it to me.)

So here's my first question:

The website of a band has a kind of a diary. The children of "diary" have the template "tour" and the children of "tour" have the template "concert". "concert" has (beside others) a textarea field for the report, an image field for photos and a text field to embed a youtube video. None is required.
An overview page should show a list of all concerts. If a field of the concert (report, photos, video) has content, a specific icon shall be shown.  Everything works fine so far, but it takes about 5 seconds to render the page with about 250 concerts in the list. Currently I solve this by simply turn on the caching, but I think 5 seconds to list 250 pages is a bit too long. (What if I had to deal with 10000 pages or more?)

It runs on a virtual server with at least 1 GHz CPU and 2 Gbyte RAM.

Below is my code (a little shortened). Has somebody an idea, which could be the inefficient part?

Thank you in advance.
 

<ul>
<?php
foreach($tours->children as $tour) {
    foreach($tour->children as $concert) { ?>
        <li><?=$concert->date?>, <?=$concert->title?>, <?=$concert->city?>
        
        <?php // Check for content, then show the icon
        if($concert->text) { echo '<img src="'.$config->urls->templates.'styles/images/icon_text.png" alt="Konzertbericht" />'; }
        if(count($concert->fotos) > 0) { echo '<img src="'.$config->urls->templates.'styles/images/icon_photos.png" alt="Fotos" />'; }
        if($concert->podcast) { echo '<img src="'.$config->urls->templates.'styles/images/icon_video.png" alt="Podcast" />'; }
        ?>

        </li>

    <?php } // END foreach($tour->children as $concert) ?>
<?php } // END foreach($tours->children as $tour) ?>
</ul>
 

[WillyC]

u shuold use.vagination with 10 - 20 concerto/page

hundredos of paginas u run out memory soon

if u.cannut u may  try

stop nested foreacher.use

$tours->find( "template=concert ") { 

edit each felds:: date city text fotos podcast

checkbox ``auto join``

save

dose this help ? it.shuld 

u.can loook 2 ``markupcache`` modulos too

in end u must use paginasnation 

other wises u site can not scale

[diogo]

Just to complete Willy's answer

http://processwire.com/api/modules/markup-pager-nav/

[renobird]

u shuold use.vagination

Killed me. Dead.

Perhaps you could also list the first 20, then "load more" via ajax on request.

Also...

Welcome Seb!

[Wanze]

Hi Seb, welcome to the world of the great ProcessWire!

I'll try to translate WillyC to English

There are two problems:

1) The code is not efficient. Your two nested for-loops load to much unused pages into memory. As you need only the concerts, go directly over the concert template:

$concerts = $pages->find('template=concert'); // Maybe sort by date or another field 

2) As already mentioned above, 250+ concerts are a lot to show to a user. Try to paginate them.

But could be that the page load already decreases a lot if you try number 1)

Cheers

[Seb]

Thanks a lot for your answers.
Meanwhile I found out, that the most expensive line is if(count($concert->fotos) > 0)...
Without it, the page renders in exeptable time. Is there a more efficient way to find out, if an image field contains at least one image? I tried if($concert->fotos->first()), but there was no noticeable difference.



Not using two nested foreach-loops leads to another problem.
The list should be interrupted by a headline when a new tour begins. (My fault, I deleted most of the markup for better legibility.)
 

<?php foreach($tours->children as $tour) { ?>
    <h1><?=$tour->title?></h1>
    <ul>
    <?php foreach($tour->children as $concert) { ?>
        <li><?=$concert->date?>, <?=$concert->title?>, <?=$concert->city?>
        
        <?php // Check for content, then show the icon
        if($concert->text) { echo '<img src="'.$config->urls->templates.'styles/images/icon_text.png" alt="Konzertbericht" />'; }
        if(count($concert->fotos) > 0) { echo '<img src="'.$config->urls->templates.'styles/images/icon_photos.png" alt="Fotos" />'; }
        if($concert->podcast) { echo '<img src="'.$config->urls->templates.'styles/images/icon_video.png" alt="Podcast" />'; }
        ?>

        </li>

    <?php } // END foreach($tour->children as $concert) ?>
    </ul>
<?php } // END foreach($tours->children as $tour) ?>

Without nested loops I would have to do something like this:
 

$concerts = $pages->find('template=concert, sort=-date');
echo "<h1>$concerts->first()->parent->title</h1><ul>";
$prev = $concerts->first();
foreach($concerts as $concert) {
    if($concert->parent->title != $prev->parent->title) {
        echo "</ul><h1>$concert->parent->title</h1><ul>";
    }
    // print concert
    $prev = $concert;
}
echo "</ul>";

I think that would make it even worse.

[Soma]

250 is just a lot. I guess it would take with only outputing titles 1-2 seconds.

And how many images are there then if count() takes so much time?

$images->count() is another one but same like count($images) I guess.

If you really need to do that for such a long list (!) you maybe better of caching it, like saving it to a text field or checkbox if there are images.

You could do it after saving a page with a simple hook. So you could check with if($p->has_images) ...

Or even cache the whole html to a textfield etc. Already lots of suggestions. Pagination wold be of course the most natural solution as suggested, but not sure what you're trying to do with a list of 250+

[diogo]

This might sound a bit crazy, but I wonder if doing this is more efficient;

if($pages->get("parent=$tour, name=$concert->name, fotos.count>0")->id)
 

[Macrura]

also, i wonder if you could set these up not hierarchically - just have concerts, tours, and diary.

then when you create a concert you just select which tour it is (page select); so this way if there is a concert that is not part of a tour it doesn't get stuck down in that page tree; depending on the diary entry format, you could have a selector on that for the tour and the concert...

the more i work with pw, the more careful i get about pigeonholing certain content heirarchies, since i've run into some issues when the client changed their mind about the relationships of their content; so now i use a lot of page field references to connect things together and less relying on the page tree...

[Seb]

Thanks again for your answers.

I tried diogo's idea and used if($pages->get("parent=$tour, name=$concert->name, fotos.count>0")->id) as condition. It's really much faster. Time for rendering reduced from average 4.89sec to 1.49sec. Good idea. Thank you.

I will also try Soma's approach of caching the number of images via an extra field. Sounds promising, too. Thank you.

In this project, I don't want to change the hierarchy because the customer is used to it now and works well with it. But in the near future I have a project, a website for a theater, with more complex dependencies where I will learn to work with page selectors.

Currently I am totally satisfied with the solution of simply turning on ProcessWire's caching function. Nevertheless I want to write efficient code, so I drew up this topic.

Thank you all for your time.

[ryan]

I tried diogo's idea and used if($pages->get("parent=$tour, name=$concert->name, fotos.count>0")->id) as condition. It's really much faster. Time for rendering reduced from average 4.89sec to 1.49sec. Good idea. Thank you.

There must be a lot of photos in there for the counting of that field to be a bottleneck? Something that may be even faster is to load the concerts that have photos in one query and label them as such, then load the concerts that have no photos in another:

$concerts = $pages->find("..., fotos.count>0");
foreach($conerts as $concert) $concert->has_photos = true; 
$concerts->add($pages->find("..., fotos.count=0"))->sort("title");  

Using that method, output code would just check $concert->has_photos; rather than count($concert->fotos); But you will have reduced 250+ additional queries down to 2. 

[ryan]

Also, make your "text", "date", "city" and "podcast" fields autojoin in your advanced field settings like WillyC mentioned. That should cut out a few hundred more queries as well.

To avoid the nested loop, just keep track of when you need to output your headline. You want your data grouped by tour, so you'd sort by something from the parent, followed by the concert date. Perhaps "sort=parent_id, sort=date" in the concert finding query. Then keep a $lastParentID variable that you set at the end of your loop ($lastParentID = $concert->parent_id. At the beginning of your loop, check if $lastParentID != $concert->parent_id and display a headline when the condition matches.