Jump to content
creativejay

List of pages by template timing out

Recommended Posts

I'm displaying a list of products which are found by their templates, but the pages are taking a very long time to load. At first, I blamed it on my image rendering (using PIM2), but even with all those images now stored in the file tree, the page is taking abysmally long to load. ProCache seems to help but I don't feel as though what I'm trying to do should be gnawing the bones of my resources quite so long.

The variable for the selector is defined in my header include:

$productCatList="prod_series|prod_series_ethernet|prod_series_access|prod_series_accessories|prod_series_fiber|prod_series_pwr_supplies|prod_series_pwr_systems|prod_series_wireless";
$getCurrentProdOptions="template=$productCatList, prod_status_pages!=1554|1559|1560|4242";

Then in the template for the page upon which the directory loads:

$products = $pages->find("$getCurrentProdOptions"); 
include_once("./prod-list-row.inc");
echo $out;

And the prod-list-row.inc foreach (which is on every page that's exhibiting the slowdown):

<?php 
	$sum = 0;
	$out ="";
	$out .= "<div class='span_12_of_12'>\n"; 
	foreach($products as $p){
		$sum += 1; 
		 	if ($sum % 2 == 0) {
		 		$bgcolor = '#fff';
			} else { $bgcolor = '#e4e4e4';}
		$par = $p->parent;
		$out .="<div class='section group' style='background: $bgcolor ; min-height: 110px'>\n";
		$img = $p->prod_image;
		$thumb = $img->pim2Load('squarethumb100')->canvas(100,100,array(0,0,0,0),'north',0)->pimSave()->url;
   		
		$out .="<div data-match-height='{$p->title}' class='col span_2_of_12 hide'>";
		
		$out .="<a href='{$p->url}'><span class='product-image-box'><img src='{$thumb}' alt='{$p->title}' title='{$p->title}'></span></a>";
		$out .= "</div>";
		$out .= "<div data-match-height='{$p->title}' class='col span_6_of_12'>";
		$out .= "<div class='prod-list-name-label'><a href='{$p->url}'>{$p->title}</a></div>";
		if($page!=$par) {
			$out .= "<div class='prod-list-category-label' style='font-size: .7em;'>Category: <a href='{$par->url}'>{$par->title}</a></div>";
			}		
		
		$out .=	"<div class='list-headline' style='font-size: .8em;'>{$p->headline}</div>";
		$out .="<div class='learn-more-buttons-sm'>";
        $out .="<a href='{$p->url}' title='Product Specs and Documentation'><span class='find-out-more-button' style='font-size: .8em;'><i style='font-size: .8em;' class='fa fa-lightbulb-o' ></i> &nbsp; Learn More</span></a>";
		$out .="</div>";
		$out .="</div> \n";
		$out .= "	<div data-match-height='{$p->title}' class='col span_4_of_12'>";
		if(count($p->prod_feat_imgs) >0 ){
			$out .= "<div class='featured-icons-list' margin: 2em .5em;'>";
			foreach($p->prod_feat_imgs as $feat){
				$icon = $pages->get("$feat->prod_featicon_pages");
				if($icon->image) {
					if($feat->prod_feat_textlang) { $icontitle = $feat->prod_feat_textlang;} else {$icontitle = $icon->title;}
				
				$out .= "<img src='".$icon->image->size(35,35,$imgOptions)->url . "' alt='" . $icontitle . "' title='" . $icontitle . "' class='listing-feat-icon' style='margin-right: .5em;' />";
				}
			}
			$out .= "</div>";
			if($p->prod_product_line){
				foreach($p->prod_product_line as $pline)
					if($pline->image) {
						$out .= "<div style='height: 35px;'>\n";
						$out .= "<img src='{$pline->image->size(75,35,$imgOptions)->url}' alt='{$pline->title}' />"; 
						$out .= "</div>";
					}
			}
			
		}
		
			$out .= "</div>";
		$out .="</div>";
	}
	$out .= "</div>";

 

Is there a clear culprit here of what I'm doing that's so stressing the system?

I turned off TracyDebugger because I saw another thread about that causing slowdown (even though I'm using the latest), but that had no effect. Every time I thought I found the culprit and commented it out, nothing changed.

Would appreciate some more eyes on this. Thank you!

ETA: prod_feat_imgs is a repeater field which contains a Page reference field (from which I pull the image and title) and a multilanguage textfield (to override the page reference title if it exists). Could that be the problem?

Edited by creativejay
more info

Share this post


Link to post
Share on other sites

Well, your code runs slow because:

  • You have multiple nested loops, each of which is querying pages, which would be slow by definition.
  • Inside every loop you're getting, resizing images, which is even slower than querying pages (on the first request, at least)
  • You seem to use repeater fields to store metadata for images, and each repeater item has page reference fields, which slows down your code even more

Points for improvement:

That said, here's an reformatted/rewritten version of your code:

<?php namespace ProcessWire; ?>
<style>
    .bg-white {
        background-color: #fff;
    }
    .bg-gray {
        background-color: #e4e4e4;
    }
    .group {
        min-height: 110px;
    }
    
    .prod-list-category-label {
        font-size: .7em;
    }
    
    .list-headline {
        font-size: .8em;
    }
    .featured-icons-list {
        margin: 2em .5em;
    }
</style>
<div class='span_12_of_12'>
<?php foreach ($products as $p): ?>
    <?php
    // $bgcolor = $sum % 2 == 0 ? '#fff' : '#e4e4e4';
    $bgClass = $sum % 2 == 0 ? 'bg-white' : 'bg-gray';

    $par = $p->parent;
    $img = $p->prod_image;
    $thumb = $img->pim2Load('squarethumb100')->canvas(100,100,array(0,0,0,0),'north',0)->pimSave()->url;

    ?>
    <?php // Replace inline styles with classes ?>
    <div class="section group <?= $bgClass ?>">
        <div class="col span_2_of_12 hide" data-match-height="<?= $p->title ?>">
            <a href="<?= $p->url ?>"><span class="product-image-box"><img src="<?= $thumb ?>" alt="<?= $p->title ?>" title="<?= $p->title ?>"></span></a>
        </div>
        <div class="col span_6_of_12" data-match-height="<?= $p->title ?>">
            <div class="prod-list-name-label"><a href="<?= $p->url ?>"><?= $p->title ?></a></div>

            <?php if ($page != $par): ?>
                <?php // Replace inline styles with classes ?>
                <div class='prod-list-category-label'>Category: <a href='<?= $par->url ?>'><?= $par->title ?></a></div>
            <?php endif; ?>
            <?php // Replace inline styles with classes ?>
            <div class="list-headline"><?= $p->headline ?></div>
            <div class="learn-more-buttons-sm">
                <a href='<?= $p->url ?>' title='Product Specs and Documentation'><span class='find-out-more-button' style='font-size: .8em;'><i style='font-size: .8em;' class='fa fa-lightbulb-o' ></i>   Learn More</span></a>
            </div>
        </div>
        <div data-match-height='<?= $p->title ?>' class='col span_4_of_12'>
            <?php if ($p->prod_feat_imgs->count > 0): ?>
            <div class='featured-icons-list'>
                <?php
                foreach ($p->prod_feat_imgs as $feat) {
                    $icon = $pages->get("$feat->prod_featicon_pages");
                    if ($icon->image) {
                        // Loop up Elvis operator
                        $icontitle = $feat->prod_feat_textlang ?: $icon->title;
                        $out .= "<img src='" . $icon->image->size(35, 35, $imgOptions)->url . "' alt='" . $icontitle . "' title='" . $icontitle . "' class='listing-feat-icon' style='margin-right: .5em;' />";
                    }
                }
                ?>
            </div>



            <?php if($p->prod_product_line){
                foreach($p->prod_product_line as $pline)
                    if($pline->image) {
                        $out .= "<div style='height: 35px;'>\n";
                        $out .= "<img src='{$pline->image->size(75,35,$imgOptions)->url}' alt='{$pline->title}' />";
                        $out .= "</div>";
                    }
            } ?>

            <?php endif; ?>
        </div>
    </div>
<?php endforeach; ?>
</div>

 

  • Like 3

Share this post


Link to post
Share on other sites

Limiting and paginating would be sensible too.

Even if the list is currently not very long, any time you list pages that a client may add to (I assume the client can add new products) you should limit and paginate because you don't know how many pages this may ultimately grow to.

  • Like 1

Share this post


Link to post
Share on other sites

Thank you both very much for taking the time to reply and help me out. abdus, thank you very much for the rewritten code. I begin to see what you mean.

13 hours ago, Robin S said:

Even if the list is currently not very long, any time you list pages that a client may add to (I assume the client can add new products) you should limit and paginate because you don't know how many pages this may ultimately grow to.

Thankfully, in this case, I am the client. I am building this site for my day job, and I know a bit about the number of products that typically are added in any given year (generally no more than 5). 

On the individual product categories, (organized by the product templates' placement in the page tree), right now the maximum number is 37. While that section will grow, my main concern is the speed of the pages upon which every product is listed for the convenience of someone (most often, myself) who needs to reference something quickly.

The site needs to launch last year (of course!), so while I figure this out I will likely keep such bloated pages unpubbed so I can work to improve their performance before I make them public.

 

I'm self-taught, and this is the most complicated (in terms of interlinking pages) that I've ever created a site. I had no idea it was going to so greatly impact performance to loop it the way I have until the information was already populated and I saw the results of the foreach. 

 

15 hours ago, abdus said:

Use Image Extra module instead of repeaters for saving extra data with images

The reason that I have the prod_feat_imgs in a Repeater field (as a page reference) is that I've got ~45 icons that appear both in the category list and on the product pages themselves. These have default value captions (in the referenced page's title field) which occasionally need to be overridden per product (for example one boasts of operating temperature range and in 85% of cases is one value (set in the 'page-select-option' page along with the icon) but in other cases will need to be overridden.

So prod_feat_imgs is a repeater which includes a page reference (prod_feat_ and a textfield to override the referenced page's title.

I have set up the product templates with a large number of getPage fields (and a whole hidden section of the page tree dedicated to universal options to choose from, for the same reason as above: future proofing).

Thankfully not that many are built into repeaters (I started to realize what a wicked web I'd woven as I tried to get the repeater information into the markup but didn't realize it taxed PW to run as much as it did me to code). Many, however, are multi-select.

In the case of prod_feat_imgs, I will periodically need to go in an update the entire list of 45 images if we redesign them. It seemed future-proof to update them on 45 pages rather than track them to separate image fields across potentially 200-300 product pages.

So, all that was to explain my thought process in why I'm not putting the Image field directly in the product template fieldset. It doesn't seem that Image Extra would help me in this use case. If you have another suggestion to how to approach this, I am open to hearing it!

Right now the best alternative I can come up with is to take the page reference (prod_feat_iconpages) out of the repeater and just add additional pages to that section of my page tree for the variations. This will become ungainly in the future, though, as cases like the temperature example would require a new entry for a minor variation in spec.

I'm planning further updates to the site (the full improvement list I intended to be ready for launch, but there was too much data entry required for the time allotted), and this image's caption could be overridden by the actual specification for temperature, for which I was planning to use a table field on prod-series' children pages, prod-model (this promises another headache of markup looping. I'm almost glad I didn't have time to include this in the current launch).

 

15 hours ago, abdus said:

Use WireCache a lot

I am using ProCache currently. I thought ProCache handled the basic functions of WireCache and extended them. Am I mistaken?

15 hours ago, abdus said:

Not a speed improvement, but escape long spreads of HTML from PHP, you don't have to write all of your HTML into strings

Thank you, I will adopt this practice going forward.

15 hours ago, abdus said:

ternary operators and elvis operator

Thank you. I did want to look into these (having seen them in sample code) but didn't have the vocabulary to search for them! (The worst part about being self-taught).

15 hours ago, abdus said:

Instead of writing CSS into HTML elements, use CSS classes and separate your CSS from HTML. You can refactor $bgcolor as .bg-white and .bg-gray, for example

Yes, thank you. I write it into the HTML elements while I'm fussing about with the templates, then migrate them over into my referenced css file.  But it would make my life easier to at least structure it the way you did in the rewritten code. 

 

15 hours ago, abdus said:

Check out this thread

Thank you. I'm not sure on the direct takeaway of that thread (I did find it before I posted this one). Are you suggesting switching to AJAX/JSON or using the MySQL monitor?

15 hours ago, abdus said:

You have multiple nested loops, each of which is querying pages, which would be slow by definition.

This is the most complicated site I've ever designed, and I was under the impression that, as long as I refined my queries down as much as I could (using get instead of find wherever possible), that the query was not a terrible drain on the process, and that this was the PW advantage over other CMS. Obviously, I need to reapproach my thinking on the matter. 

15 hours ago, abdus said:

Inside every loop you're getting, resizing images, which is even slower than querying pages (on the first request, at least)

Yes, and this was my red herring. As I adjusted image sizes in the template markup, I fully expected the delay. It was only when I finished and the page was still slow that I realized there was another, probably more fundamental, problem with my coding.

 

 

 

page tree down to page-select-options level.png

prod_feat_imgs in product-series page.png

page tree down to produc-series template.png

Share this post


Link to post
Share on other sites

Adding to what @abdus said: make sure you haven't got a name resolution issue that slows down database access (it often helps setting 127.0.0.1 instead of localhost for $config->dbHost to work around IPv6 issues). If you still want an additional boost, consider enabling opcode caching (e.g. with PHP's opcache module).

  • Like 4

Share this post


Link to post
Share on other sites

My gut feeling is that the Page Reference fields, repeaters, foreach nesting is not what is causing this slowness. For a listing of only 37 product pages there should not be that much of a delay. (How long is "abysmally slow", by the way?) I'm doing similar things for with page-per-image setups (although with PageTable rather than repeater) with much larger numbers of pages and there isn't a significant impact on speed. And for pages in the ProCache cache there should be no impact whatsoever.

So I think you have some other issue going on. Not sure what though, sorry. If you're developing locally, try testing the site on a remote server and see if you experience the same slowness. Or vice-versa if you are developing remotely.

Also: start commenting/cutting out blocks of your template code to try and isolate if one area is responsible.

  • Like 1

Share this post


Link to post
Share on other sites
40 minutes ago, Robin S said:

My gut feeling is that the Page Reference fields, repeaters, foreach nesting is not what is causing this slowness. For a listing of only 37 product pages there should not be that much of a delay. (How long is "abysmally slow", by the way?) I'm doing similar things for with page-per-image setups (although with PageTable rather than repeater) with much larger numbers of pages and there isn't a significant impact on speed. And for pages in the ProCache cache there should be no impact whatsoever.

So I think you have some other issue going on. Not sure what though, sorry. If you're developing locally, try testing the site on a remote server and see if you experience the same slowness. Or vice-versa if you are developing remotely.

Also: start commenting/cutting out blocks of your template code to try and isolate if one area is responsible.

 

Eliminating the prod_feat_imgs call from the foreach did seem to speed things up, but not to the near-instant page loads I've come to expect. I really hope I can improve performance from the scripting and that I don't need to find a new host. I'm developing on a managed VPS account, in a subdirectory.

To answer your question, abysmally slow was 26 seconds or more per page (and if the images were being rendered afresh, up to four minutes, including a time-out and a reload to finish processing them).

Share this post


Link to post
Share on other sites

You can use

$start = microtime(true);
// code to test
echo microtime(true) - $start;
// or log the time
$log->error("code xyz took: " . microtime(true) - $start);

to find which part takes the most time to execute.

http://php.net/microtime

  • Like 1

Share this post


Link to post
Share on other sites
2 hours ago, fbg13 said:

You can use


$start = microtime(true);
// code to test
echo microtime(true) - $start;
// or log the time
$log->error("code xyz took: " . microtime(true) - $start);

to find which part takes the most time to execute.

http://php.net/microtime

Also, check out the Performance Panel in Tracy to show the time between named breakpoints.
https://processwire.com/blog/posts/introducing-tracy-debugger/#performance-panel

  • Like 2

Share this post


Link to post
Share on other sites
8 hours ago, creativejay said:

I'm developing on a managed VPS account, in a subdirectory.

I always put all my sites in a subdirectory so I do not think it should matter regarding the performance of the site. I also have a site on a managed VPS (WHM) and originally Tracy reported about 500ms execution time for its 10 product listing page. I asked the support to optimize the VPS – originally it was in its default setup state – and they managed to speed it up so right now the same page renders in about 130ms. They changed php settings, etc... We did not discuss in detail what they had done, I was just happy to see the result ;) 

I'm not saying fine tuning your VPS will surely solve your problem but it might help too.

Edited by szabesz
typo
  • Like 1

Share this post


Link to post
Share on other sites
8 hours ago, szabesz said:

I always put all my sites in a subdirectory so I do not think it should matter regarding the performance of the site. I also have a site on a managed VPS (WHM) and originally Tracy reported about 500ms execution time for its 10 product listing page. I asked the support to optimize the VPS – originally it was in its default setup state – and they managed to speed it up so right now the same page renders in about 130ms. They changed php settings, etc... We did not discuss in detail what they had done, I was just happy to see the result ;) 

I'm not saying fine tuning your VPS will surely solve your problem but it might help too.

Thank you szabesz, I asked my host about it and the response was "what did you have in mind?" :P

They did offer to install litespeed and provided this information on what it offers: https://www.a2hosting.com/kb/a2-hosting-products/turbo-web-hosting/differences-between-turbo-and-swift-servers

To those of you who understand this better than I, would this be worthwhile for a monthly fee, or are these largely improvements I can handle with PWire modules?

I will play around with the timestamps this morning now that my files are in their final resting place, before I must throw the switch later today.

  • Like 1

Share this post


Link to post
Share on other sites
14 minutes ago, creativejay said:

They did offer to install litespeed and provided this information on what it offers: https://www.a2hosting.com/kb/a2-hosting-products/turbo-web-hosting/differences-between-turbo-and-swift-servers

To those of you who understand this better than I, would this be worthwhile for a monthly fee, or are these largely improvements I can handle with PWire modules?

ProCache should cover almost all of what they offer (mostly caching). I'd say it's a cash grab.

You said you were hosting your websites on a managed VPS, how is your linux skills? Try switching to another hoster. I am a happy user of Vultr and DigitalOcean, and never had problems with them. They bill (IIRC) hourly, so you can try it for a few days for a few bucks and see if it works out for you.

Managing your own VPS isn't too difficult. As long as you dont keep many ports open (80, 443 for web and 22 or a different port for SSH), set up a fail2ban and you'd be mostly OK in terms of security, and you get to extend your skillset along the way.

  • Like 2

Share this post


Link to post
Share on other sites

Oh, okay a new reply without my prompting (I do love this host's support response):

Quote

You are currently running PHP 5 with dso | Ruid2. I would suggest that we install PHP 7 or multiple versions 5-7 if needed so you can take advantage of performance increases. 

Anything I need to be aware of if I have them install PHP 7?

Share this post


Link to post
Share on other sites

Well

12 minutes ago, creativejay said:

"what did you have in mind?"

Well, no comment....

I looked up the conversation with my VPS team and that's all the "depth" we talked about the matter: "I did however make some performance optimizations to your server to improve overall performance, such as switching the site to use PHP-FPM as the PHP handler, which is much faster than the default handler."

It's a managed VPS and as long as it is speedy and works without issues, I am happy.

5 minutes ago, creativejay said:

Anything I need to be aware of if I have them install PHP 7?

Switching to PHP 7.0.x should be okay even if you have 3rd party modules installed which work with ProcessWire 3.x. I'm using it too for all my sites.

Edited by szabesz
typo
  • Like 1

Share this post


Link to post
Share on other sites
5 minutes ago, abdus said:

how is your linux skills?

That's the cat with the short tail, right?  XD

 

(In other words, I have zero linux skills)

  • Like 1

Share this post


Link to post
Share on other sites

Checking times in Tracy, I see (not that surprising) that my selector for locating the products on the site is a bit of a hog:

Quote
template=prod_series|prod_series_ethernet|prod_series_access|prod_series_accessories|prod_series_fiber|prod_series_pwr_supplies|prod_series_pwr_systems|prod_series_wireless, prod_status_pages!=1554|1559|1560|4242   1294.8

Not surprising, since a lot of the pages that hit this slowdown contain a similar $pages->find() call.

Which is the largest offender here? That it's returning 245 results, or that it would technically be checking every page in the tree for that template and status?

I added a has_parent=1048 to confine the search to the Products section, but that only improved the timestamp to 1240.6

Share this post


Link to post
Share on other sites

Try caching the result of that

$selector = 'template=prod_series|prod_series_ethernet|prod_series_access|prod_series_accessories|prod_series_fiber|prod_series_pwr_supplies|prod_series_pwr_systems|prod_series_wireless, prod_status_pages!=1554|1559|1560|4242';
if (!($list = $cache->get('products-main-view', WireCache::expireHourly))) {
    $list = $pages->find($selector);
    $cache->save('products-main-view', $list);
}

// do sth with the $list

PW will save a list of ids (such as 3545|2556|1456|7889|3456) to caches table in DB and it wont have to manually build that list at every non-cached request (I keep forgetting that you're using ProCache). Once it's cached, it'll retrieve pages back using IDs and you can't get faster than that.

Procache is disabled when you're logged in. Are your results from an incognito window?

Share this post


Link to post
Share on other sites
6 minutes ago, abdus said:

Are your results from an incognito window?

No, I am using the TracyDebugger bar that loads for the superuser.

I implemented that code and the first instance took the expected 1240 ms, while the second instance (the original call on the page) then took .6 ms. The pages still load somewhat slowly in the second browser I have open (where I am not logged in).

8 minutes ago, abdus said:

I keep forgetting that you're using ProCache.

I currently have ProCache disabled while I try to suss this out. Does this mean ProCache will provide similar caching to the code you offered, or should I still go through and force cache the common id lists?

Share this post


Link to post
Share on other sites
1 minute ago, creativejay said:

No, I am using the TracyDebugger bar that loads for the superuser.

Just a quick FYI - you can get Tracy's debug bar when logged out two different ways. If you are on a local dev machine, check the "Force Guest Users into Development Mode on Localhost" option. The other way if you are on a live server is to use the User Switcher to logout.

Hope that helps you get data you may want while logged out.

  • Like 3

Share this post


Link to post
Share on other sites

Okay, my host is prepared to enable PHP-FPM on my say-so. PHP has been upgraded though the active version is currently 5.6.

Is there an ideal PHP version, for the current release of PW3.x?

And should I be aware of anything before requesting PHP-FPM be enabled? I've been asked "if I'm sure" and that's giving me heart palpitations (that and all the coffee that kept me going through this mess all weekend).

Share this post


Link to post
Share on other sites
38 minutes ago, creativejay said:

PHP has been upgraded though the active version is currently 5.6.

Definitely make the switch to PHP 7.0 or 7.1

  • Like 2

Share this post


Link to post
Share on other sites
17 minutes ago, creativejay said:

And should I be aware of anything before requesting PHP-FPM be enabled? I've been asked "if I'm sure" and that's giving me heart palpitations (that and all the coffee that kept me going through this mess all weekend).

I'm no expert at all but they should be able to switch back should anything go awry. I do not have any issue with it...

5 minutes ago, abdus said:

PHP 7.0 of 7.1

I guess there is a typo: 7.0 or 7.1
I would start with 7.0 and you should see improvement. Afterwards you might want to try 7.1 too, of course,

  • Like 1

Share this post


Link to post
Share on other sites

Thanks everyone! I wouldn't have gotten this far without you.

I have asked them to switch to 7.0. I will test and then ask them to enable PHP-FPM, test again.

Then switch TracyDebugger into Production Mode, then enable ProCache. Then give the site a last lookie-loo, and then throw the switch.

My eyes hurt!

ETA: Without any of the above being completed, the site is loading faster for a guest. A very good sign!

Edited by creativejay
noticed speed increase while waiting!
  • Like 1

Share this post


Link to post
Share on other sites

Even faster in 7.0, but now we've got some PHP errors getting thrown.

Does anyone happen to know what modules, aside from pdo/mysql, might be required to be installed on the server for ProcessWire?

Share this post


Link to post
Share on other sites
5 minutes ago, creativejay said:

PHP errors getting thrown.

I recommend showing us the errors to track things down more easily. If you have long error output strings, you might want to use the forum's "spoiler" feature to make your post shorter.

Edited by szabesz
typo
  • Like 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By DooM
      Hello guys,
      I'm trying to figure out how to sync fields and templates between staging and production environments.
      I've found Migrations module by Lostkobrakai, but with use of it all the fields and templates must be created by API, which is kind of uncomfortable.
      I also tried ProcessDatabaseBackups module which can export only certain tables, but I don't think it's the best practice to do that.
      How do you guys solve this problem? It's very annoying to setup everything three times (dev, staging, production).
      Thanks a lot :)
    • By Brawlz
      Hi,
      I hope this is the correct section for my problem.
      All I need is a connection to an external Database and a query gettings some data. I do this in a processwire Page-Template. I am honestly not sure if it is a problem with processwire or my code:
      $host = ‚XXXXX’; $user = ‚XXXXX‘; $pass = ‚XXXXX‘; $db = ‚XXXXX‘; $port = ‚3306‘; $mydb = new Database($host, $user, $pass, $db , $port);  $result = $mydb->query("SELECT * FROM char“);  while($row = $result->fetch_assoc()) {  print_r($row);  }  
      Produces the following error:
      Error: Exception: DB connect error 2002 - Connection timed out (in /customers/9/4/e/XXXX.de/httpd.www/wire/core/Database.php line 79)
       
      I also tried connecting without the $port variable but got the same error.
    • By Mobiletrooper
      Hey Ryan, hey friends,
      we, Mobile Trooper a digital agency based in Germany, use ProcessWire for an Enterprise-grade Intranet publishing portal which is under heavy development for over 3 years now. Over the years not only the user base grew but also the platform in general. We introduced lots and lots of features thanks to ProcessWire's absurd flexibility. We came along many CMS (or CMFs for that matter) that don't even come close to ProcessWire. Closest we came across was Locomotive (Rails-based) and Pimcore (PHP based).
      So this is not your typical ProcessWire installation in terms of size.
      Currently we count:
      140 Templates (Some have 1 page, some have >6000 pages)
      313 Fields
      ~ 15k Users (For an intranet portal? That's heavy.)
      ~ 195 431 Pages (At least that's the current AUTOINCREMENT)
       
      I think we came to a point where ProcessWire isn't as scalable anymore as it used to be. Our latest research measured over 20 seconds of load time (the time PHP spent scambling the HTML together). That's unacceptable unfortunately. We've implemented common performance strategies like:
      We're running on fat machines (DB server has 32 gigs RAM, Prod Web server has 32gigs as well. Both are running on quadcores (xeons) hosted by Azure.
      We have load balancing in place, but still, a single server needs up to 20 sec to respond to a single request averaging at around about 12 sec.
      In our research we came across pages that sent over 1000 SQL queries with lots of JOINs. This is obviously needed because of PWs architecture (a field a table) but does this slow mySQL down much? For the start page we need to get somewhere around 60-80 pages, each page needs to be queried for ~12 fields to be displayed correctly, is this too much? There are many different fields involved like multiple Page-fields which hold tags, categories etc.
      We installed Profiler Pro but it does not seem to show us the real bottleneck, it just says that everything is kinda slow and sums up to the grand total we mentioned above.
      ProCache does not help us because every user is seeing something different, so we can cache some fragments but they usually measure at around 10ms. We can't spend time optimising if we can't expect an affordable benefit. Therefore we opted against ProCache and used our own module which generates these cache fragments lazily. 
      That speeds up the whole page rendering to ~7 sec, this is acceptable compared to 20sec but still ridiculously long.
      Our page consists of mainly dynamic parts changing every 2-5 minutes. It's different across multiple users based on their location, language and other preferences.
      We also have about 120 people working on the processwire backend the whole day concurrently.
       
      What do you guys think?
      Here are my questions, hopefully we can collect these in a wiki or something because I'm sure more and more people will hit that break sooner than they hoped they would:
       
      - Should we opt for optimising the database? Since >2k per request is a lot even for a mysql server, webserver cpu is basically idling at that time.
      - Do you think at this point it makes sense to use ProcessWire as a simple REST API?
      - In your experience, what fieldtypes are expensive? Page? RepeaterMatrix?
      - Ryan, what do you consider as the primary bottleneck of processwire?
      - Is the amount of fields too much? Would it be better if we would try to reuse fields as much as possible?
      - Is there an option to hook onto ProcessWires SQL builder? So we can write custom SQL for some selectors?
       
      Thanks and lots of wishes,
      Pascal from Mobile Trooper
       
       
    • By Noel Boss
      ProcessWire & Vue.js — a Lovestory
      Introducing the all new ICF Conference Website
        The new ICF Conference Page — Fearless
      » What would happen if we were equipped to fearlessly face the daily challenges and live a life without fear? «
      This question is at the core of our next ICF Conference in 2019 in Zurich. Its also the question we set out to answer in terms of developing the new website; the all new ICF Conference website is our most advanced website in terms of technology, designed to take advantage of the latest web-technologies.
      Its a brand new design powered by a lean setup, using ProcessWire for easy content management and a slick frontend based on Vue.js, Quasar and a heavily customized Uikit theme.
        Technology-stack — From backend to frontend, technologies that are fun, easy and fast to develop with We built on the ICF Ladieslounge website as a solid foundation and took our learnings from building our last Conference Booklet PWA (Progressive Web App) and applied it to the new website.
      Some highlights of the new ICF Conference website:
      Completely decoupled backend and frontend Custom design based on Uikit frontend framework Changing of languages happens instantly, no page-reload required Easy content updates thanks to ProcessWire All data is transferred using a single request returning custom JSON



      » Continue reading on Medium
      And please don't forget to clap and share: 

       
    • By Noel Boss
      Page Query Boss
      Build complex nested queries containing multiple fields and pages and return an array or JSON. This is useful to fetch data for SPA and PWA.
      You can use the Module to transform a ProcessWire Page or PageArray – even RepeaterMatrixPageArrays – into an array or JSON. Queries can be nested and contain closures as callback functions. Some field-types are transformed automatically, like Pageimages or MapMarker.
      Installation
      Via ProcessWire Backend
      It is recommended to install the Module via the ProcessWire admin "Modules" > "Site" > "Add New" > "Add Module from Directory" using the PageQueryBoss class name.
      Manually
      Download the files from Github or the ProcessWire repository: https://modules.processwire.com/modules/page-query-builder/
      Copy all of the files for this module into /site/modules/PageQueryBoss/ Go to “Modules > Refresh” in your admin, and then click “install” for the this module. Module Methods
      There are two main methods:
      Return query as JSON
      $page->pageQueryJson($query); Return query as Array
      $page->pageQueryArray($query); Building the query
      The query can contain key and value pairs, or only keys. It can be nested and 
      contain closures for dynamic values. To illustrate a short example:
      // simple query: $query = [ 'height', 'floors', ]; $pages->find('template=skyscraper')->pageQueryJson($query); Queries can be nested, contain page names, template names or contain functions and ProcessWire selectors:
      // simple query: $query = [ 'height', 'floors', 'images', // < some fileds contain default sub-queries to return data 'files' => [ // but you can also overrdide these defaults: 'filename' 'ext', 'url', ], // Assuming there are child pages with the architec template, or a // field name with a page relation to architects 'architect' => [ // sub-query 'name', 'email' ], // queries can contain closure functions that return dynamic content 'querytime' => function($parent){ return "Query for $parent->title was built ".time(); } ]; $pages->find('template=skyscraper')->pageQueryJson($query); Keys:
      A single fieldname; height or floors or architects 
      The Module can handle the following fields:
      Strings, Dates, Integer… any default one-dimensional value Page references Pageimages Pagefiles PageArray MapMarker FieldtypeFunctional A template name; skyscraper or city
      Name of a child page (page.child.name=pagename); my-page-name A ProcessWire selector; template=building, floors>=25
      A new name for the returned index passed by a # delimiter:
      // the field skyscraper will be renamed to "building": $query = ["skyscraper`#building`"]  
      Key value pars:
      Any of the keys above (1-5) with an new nested sub-query array:
      $query = [ 'skyscraper' => [ 'height', 'floors' ], 'architect' => [ 'title', 'email' ], ]  
      A named key and a closure function to process and return a query. The closure gets the parent object as argument:
      $query = [ 'architecs' => function($parent) { $architects = $parent->find('template=architect'); return $architects->arrayQuery(['name', 'email']); // or return $architects->explode('name, email'); } ] Real life example:
      $query = [ 'title', 'subtitle', // naming the key invitation 'template=Invitation, limit=1#invitation' => [ 'title', 'subtitle', 'body', ], // returns global speakers and local ones... 'speakers' => function($page){ $speakers = $page->speaker_relation; $speakers = $speakers->prepend(wire('pages')->find('template=Speaker, global=1, sort=-id')); // build a query of the speakers with return $speakers->arrayQuery([ 'title#name', // rename title field to name 'subtitle#ministry', // rename subtitle field to ministry 'links' => [ 'linklabel#label', // rename linklabel field to minlabelistry 'link' ], ]); }, 'Program' => [ // Child Pages with template=Program 'title', 'summary', 'start' => function($parent){ // calculate the startdate from timetables return $parent->children->first->date; }, 'end' => function($parent){ // calculate the endate from timetables return $parent->children->last->date; }, 'Timetable' => [ 'date', // date 'timetable#entry'=> [ 'time#start', // time 'time_until#end', // time 'subtitle#description', // entry title ], ], ], // ProcessWire selector, selecting children > name result "location" 'template=Location, limit=1#location' => [ 'title#city', // summary title field to city 'body', 'country', 'venue', 'summary#address', // rename summary field to address 'link#tickets', // rename ticket link 'map', // Mapmarker field, automatically transformed 'images', 'infos#categories' => [ // repeater matrix! > rename to categories 'title#name', // rename title field to name 'entries' => [ // nested repeater matrix! 'title', 'body' ] ], ], ]; if ($input->urlSegment1 === 'json') { header('Content-type: application/json'); echo $page->pageQueryJson($query); exit(); } Module default settings
      The modules settings are public. They can be directly modified, for example:
      $modules->get('PageQueryBoss')->debug = true; $modules->get('PageQueryBoss')->defaults = []; // reset all defaults Default queries for fields:
      Some field-types or templates come with default selectors, like Pageimages etc. These are the default queries:
      // Access and modify default queries: $modules->get('PageQueryBoss')->defaults['queries'] … public $defaults = [ 'queries' => [ 'Pageimages' => [ 'basename', 'url', 'httpUrl', 'description', 'ext', 'focus', ], 'Pagefiles' => [ 'basename', 'url', 'httpUrl', 'description', 'ext', 'filesize', 'filesizeStr', 'hash', ], 'MapMarker' => [ 'lat', 'lng', 'zoom', 'address', ], 'User' => [ 'name', 'email', ], ], ]; These defaults will only be used if there is no nested sub-query for the respective type. If you query a field with complex data and do not provide a sub-query, it will be transformed accordingly:
      $page->pageQueryArry(['images']); // returns something like this 'images' => [ 'basename', 'url', 'httpUrl', 'description', 'ext', 'focus'=> [ 'top', 'left', 'zoom', 'default', 'str', ] ]; You can always provide your own sub-query, so the defaults will not be used:
      $page->pageQueryArry([ 'images' => [ 'filename', 'description' ], ]); Overriding default queries:
      You can also override the defaults, for example
      $modules->get('PageQueryBoss')->defaults['queries']['Pageimages'] = [ 'basename', 'url', 'description', ]; Index of nested elements
      The index for nested elements can be adjusted. This is also done with defaults. There are 3 possibilities:
      Nested by name (default) Nested by ID Nested by numerical index Named index (default):
      This is the default setting. If you have a field that contains sub-items, the name will be the key in the results:
      // example $pagesByName = [ 'page-1-name' => [ 'title' => "Page one title", 'name' => 'page-1-name', ], 'page-2-name' => [ 'title' => "Page two title", 'name' => 'page-2-name', ] ] ID based index:
      If an object is listed in $defaults['index-id'] the id will be the key in the results. Currently, no items are listed as defaults for id-based index:
      // Set pages to get ID based index: $modules->get('PageQueryBoss')->defaults['index-id']['Page']; // Example return array: $pagesById = [ 123 => [ 'title' => "Page one title", 'name' => 123, ], 124 => [ 'title' => "Page two title", 'name' => 124, ] ] Number based index
      By default, a couple of fields are transformed automatically to contain numbered indexes:
      // objects or template names that should use numerical indexes for children instead of names $defaults['index-n'] => [ 'Pageimage', 'Pagefile', 'RepeaterMatrixPage', ]; // example $images = [ 0 => [ 'filename' => "image1.jpg", ], 1 => [ 'filename' => "image2.jpg", ] ] Tipp: When you remove the key 'Pageimage' from $defaults['index-n'], the index will again be name-based.
       
      Help-fill closures & tipps:
      These are few helpfill closure functions you might want to use or could help as a
      starting point for your own (let me know if you have your own):

      Get an overview of languages:
          $query = ['languages' => function($page){         $ar = [];         $l=0;         foreach (wire('languages') as $language) {             // build the json url with segment 1             $ar[$l]['url']= $page->localHttpUrl($language).wire('input')->urlSegment1;             $ar[$l]['name'] = $language->name == 'default' ? 'en' : $language->name;             $ar[$l]['title'] = $language->getLanguageValue($language, 'title');             $ar[$l]['active'] = $language->id == wire('user')->language->id;             $l++;         }         return $ar;     }]; Get county info from ContinentsAndCountries Module
      Using the [ContinentsAndCountries Module](https://modules.processwire.com/modules/continents-and-countries/) you can extract iso
      code and names for countries:
          $query = ['country' => function($page){         $c = wire('modules')->get('ContinentsAndCountries')->findBy('countries', array('name', 'iso', 'code'),['code' =>$page->country]);         return count($c) ? (array) $c[count($c)-1] : null;     }]; Custom strings from a RepeaterTable for interface
      Using a RepeaterMatrix you can create template string for your frontend. This is
      usefull for buttons, labels etc. The following code uses a repeater with the
      name `strings` has a `key` and a `body` field, the returned array contains the `key` field as,
      you guess, keys and the `body` field as values:
          // build custom translations     $query = ['strings' => function($page){         return array_column($page->get('strings')->each(['key', 'body']), 'body', 'key');     }]; Multilanguage with default language fallback
      Using the following setup you can handle multilanguage and return your default
      language if the requested language does not exist. The url is composed like so:
      `page/path/{language}/{content-type}` for example: `api/icf/zurich/conference/2019/de/json`
       
          // get contenttype and language (or default language if not exists)     $lang = wire('languages')->get($input->urlSegment1);     if(!$lang instanceof Nullpage){         $user->language = $lang;     } else {         $lang = $user->language;     }     // contenttype segment 2 or 1 if language not present     $contenttype = $input->urlSegment2 ? $input->urlSegment2 : $input->urlSegment1;     if ($contenttype === 'json') {         header('Content-type: application/json');         echo $page->pageQueryJson($query);         exit();     } Debug
      The module respects wire('config')->debug. It integrates with TracyDebug. You can override it like so:
      // turns on debug output no mather what: $modules->get('PageQueryBoss')->debug = true; Todos
      Make defaults configurable via Backend. How could that be done in style with the default queries?
      Module in alpha Stage: Subject to change
      This module is in alpha stage … Query behaviour (especially selecting child-templates, renaming, naming etc)  could change
×
×
  • Create New...