Jump to content
sodesign

Very slow boot time - please help me find the cause!

Recommended Posts

Hello,

One of our sites is suffering from very slow boot times, and I'm not sure how to diagnose the problem.

Here's a grab of the debug panel in Tracy debugger after loading the homepage.

image.thumb.png.23f0eb30ed1baf5f0018f98900168388.png

A have a couple of questions -

  • Are all of the times listed separate items, or are some of them a breakdown? I ask because the number shown in the tracy debug bar is the total of all of the items but the wording suggests boot.load.modules, boot.load.fields etc are a breakdown of the boot.load.
  • How do I find out what these times consist of?

Currently, when using the site, and when running page speed tools, the server load time is consistently upwards of 1s often above 1.5s. This is before the browser even starts downloading resources - a quick grab from my firefox dev tools was even worse:

image.png.dd52293426e7984d2aa955d033b2acff.png

I would appreciate any advice on finding the cause here.

A few details:

  • Server is a digital ocean droplet (2GB memory + 2CPUs) running nginx and php7.0 - neither memory or cpu seem particularly taxed
  • Site has 8 locales
  • Using template cache and wirecache for heavy pieces of markup
  • We're on the latest dev branch - the speed issue has been present for the last couple of versions.
  • The speed is similar on when running locally (similar but stripped back nginx config)

 

Thanks,
Tom

image.png

Share this post


Link to post
Share on other sites

The number of hooks and PDO-queries seems rather high.

Did you modify anything (add functionality, change template code...) since you've noticed a slowing down of your site?

Did you turn off debug-mode on your live-site? It really should be.

It could be a number of reasons. Perhaps some modules are autoloaded when not even necessary. Or the way you've built it is not optimized enough. As you probably know - with PW - you can do anything in so many different ways, but some ways of doing stuff is better when you're dealing with lots of content + users. Since a few versions, we have e.g. methods like findIds() or findMany() that can speed things up.

Also, it's a good idea to check if there are new module updates available. Maybe some of them are quite old, and would benefit from an update, performance-wise.

If you need a quick fix, I'd suggest getting the ProCache module. (Perhaps there's a "x days money-back guarantee" that Ryan offers - I'm not sure.

 

 

 

Edited by dragan
slight rewording / fixing typo

Share this post


Link to post
Share on other sites

Thanks for your reply -

I've been making changes quite frequently - it's a fairly big site to which we're adding a shop. There's no one change which stands out as a huge change.

Debug mode is off on the live site, other than when temporarily testing, and all of the modules are pretty much up to date.

The load times aren't much better when in the admin which is why I haven't spent too much time trying to optimize templates - does template logic affect load time in the admin?

I don't think Procache supports nginx. I was using fastcgi_cache but it gets a little messy with the way we're handling locale redirects - we need to refine our config.

I'd rather find out the reason for the long load times before resorting to a flat cache.

Do these timings for loading /pw/page look reasonable to you?

image.thumb.png.eb06173ace7d7fa406c7368ae2dbad92.png

 

Share this post


Link to post
Share on other sites

Actually something interesting to note is how slow the Debug Panel actually is: 262ms. Typically mine shows somewhere between 15 and 35ms. Not sure if that's a contributor on your site, or indicative of the problem.

Share this post


Link to post
Share on other sites

idk, but I would try to install it on a local Apache environment instead of nginx (AMPPS, Laragon XAMPP, MAMP... what have you)

On 15.3.2018 at 7:01 PM, sodesign said:

does template logic affect load time in the admin?

yes and no. Depending on your page/tpl/field settings, some configs may run faster in page-edit-mode than others (you know, stuff like load only when opening field via AJAX etc. - but that's not TTFB, but mainly the browser needing time to fetch and render all the stuff) - but not the overall speed of moving around in the backend.

If some of your recent changes involved using / adding nested Matrix Repeaters A LOT, then this could be an issue.

 

 

Share this post


Link to post
Share on other sites

Just to follow up on this - I bought the Profiler Pro module and have been discussing this in the module forum there.

As part of this I created a test template which just outputs a string, with init.php and main.php prepending disabled - the Profiler Pro time for this is down to under 300ms, while the load time I see in my dev tools is still between 900ms and 3000ms.

My next steps are trying to find out why this discrepancy exists - very strange!

Share this post


Link to post
Share on other sites
On 3/16/2018 at 8:56 PM, LostKobrakai said:

What's the number of fields/templates?

Hi LostKobrakai, another example.

PW 3.0.131, 26 templates, 16 fields, on the front-end with Tracy,

Category page with just one field, template file empty:
Execution time 341.5 ms
CPU usage user + system 9 % + 1 %
Peak of allocated memory 14.70 MB
Included files 149
PDO Queries ($database) 57
Selector Queries 7

Article page with 11 fields, template file empty:
Execution time 338.3 ms
CPU usage user + system 11 % + 0 %
Peak of allocated memory 14.40 MB
Included files 149
PDO Queries ($database) 85
Selector Queries 7

(On a dedicated server with 8 GB RAM, PHP 7.2, MySQL 5.7, Apache.) I have no idea what could be done to reduce the number of queries and the CPU and memory usage needed to boot PW. Maybe we could reduce the number of fields, but 16 isn't a big number and it's just a set to start, and it would be difficult to reduce the number of templates.

Of course, that's not a problem with small to medium sized and low traffic websites, but it seems to limit the scalability. And yes, with low traffic and a few million pages this shouldn't be a problem either, or with 1000 pages and high traffic. But I'm not sure how we could use PW with a few million URLs and e.g. 100 page views per second (peak). And I really would be very glad to become more optimistic in this regard.

Apart from the number of fields and templates, are there any other vectors to optimize regarding the server load on boot? Maybe it would be helpful to collect some best practice techniques to improve the scalability of PW besides optimizations of templates or caching.

Share this post


Link to post
Share on other sites

The number of fields/templates should only become a problem for like 100+ numbers. 26 and 16 is certainly not in that realm.

Share this post


Link to post
Share on other sites
Execution time 166.5 ms
CPU usage user + system 20 % + 1 %
Peak of allocated memory 9.55 MB
Included files 230
OPcache 100% cached
Classes + interfaces + traits 238 + 28 + 0
Selector queries 17
HTTP method / response code GET / 200
PHP 7.2.18
Tracy 2.5.6
Server Apache/2.4

number of fields in this template: 40 (overall, maybe 70)

Tracy enabled + debug mode on

I would maybe take a look at some server-level optimizations. Or try the same install on another environment.

 

Share this post


Link to post
Share on other sites

Thank you LostKobrakai and dragan, I will have a look on a different server. (There's an older ExpressionEngine installation with an earlier version of the site on the same server, showing far less db queries, lower CPU usage and slightly less memory usage on pages with all the EE tags placed, with several loops from different channels and entries relations. So I was surprised to see this results for completely empty templates.)

dragan, how many PDO Queries ($database) did Tracy count?

Share this post


Link to post
Share on other sites

Some thoughts...

5 hours ago, Lutz Heckelmann said:

template file empty

What about any auto-prepended _init.php and/or auto-appended _main.php - any code in there that would impact performance?

5 hours ago, Lutz Heckelmann said:

with Tracy

Bear in mind that the PW debug mode and Tracy Debugger have their own memory usage, DB queries, etc. The Tracy overhead in particular could vary a lot depending on what panels are in use. To get a fair comparison across the EE and PW sites on the server you should disable PW debug mode and Tracy and use some consistent performance measurement technique/tool on both sites. 

Share this post


Link to post
Share on other sites

@Zeka Wow, 502 PDO queries, thank you. Well, the numbers of database queries shouldn't be the decisive factor, but under certain conditions those higher numbers maybe could become a problem, I fear.

Share this post


Link to post
Share on other sites

@Robin S Those results are some with auto-prepend (_init.php) and auto-appended _main.php (both empty), but they were nearly identical without.

Tracy: yes, this especially was my hope, that those additional queries etc. could be a factor. I will continue testing.

Share this post


Link to post
Share on other sites
34 minutes ago, Lutz Heckelmann said:

dragan, how many PDO Queries ($database) did Tracy count?

180

I'm too tired right now to look at each and every single query, but I'm puzzled by a few randomly spotted queries such as

SELECT false AS isLoaded, pages.templates_id AS templates_id, pages.*, pages_sortfields.sortfield, (SELECT COUNT(*) FROM pages AS children WHERE children.parent_id=pages.id) AS numChildren,field_email.data AS `email__data` 
FROM `pages` 
LEFT JOIN pages_sortfields ON pages_sortfields.pages_id=pages.id 
LEFT JOIN field_email ON field_email.pages_id=pages.id 
WHERE pages.templates_id=3 
AND pages.id IN(40) 
GROUP BY pages.id

I don't need any email field in this page / template. So I wonder why is something like this even queried? I'm sure (or I hope) there's a good reason why this query is listed, but I'm just not sure why. (by Tracy, or PW core? or just PW core-while-in-debug-mode?)

Still - to me, those timings are quite acceptable. And yes, if debug is off, it's even faster. (though I haven't set up a script to measure that - I just look at Google Chrome's performance audit results or similar for TTFB).

  • Like 1

Share this post


Link to post
Share on other sites

@dragan Many thanks. I think the timings strongly depend on the conditions. Sometimes a large number of pages could be uncached, and I know from experience that peaks (e.g. 60+ page views per second) can change the meaning of overhead enormously.

Maybe it's interesting to have a closer look on all those queries, thanks for the example. I too see queries that I don't understand so far. For example

37  SELECT id, templates_id, parent_id FROM pages WHERE (name=:name) LIMIT 2
38  SELECT id, templates_id, parent_id FROM pages WHERE (name=:name) LIMIT 2
39  SELECT id, templates_id, parent_id FROM pages WHERE (name=:name) LIMIT 2
40  SELECT id, templates_id, parent_id FROM pages WHERE (name=:name) LIMIT 2

or

  SELECT false AS isLoaded, pages.templates_id AS templates_id, pages.*, pages_sortfields.sortfield, (SELECT COUNT(*) FROM pages AS children WHERE children.parent_id=pages.id) AS numChildren,field_title.data AS `title__data` FROM `pages` LEFT JOIN pages_sortfields ON pages_sortfields.pages_id=pages.id LEFT JOIN field_title ON field_title.pages_id=pages.id WHERE pages.parent_id=1018 AND pages.templates_id=54 AND pages.id IN(1061) GROUP BY pages.id

 I will have a look at ProfilerPro, maybe it could help to get a better understanding of the boot process.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Mobiletrooper
      Hey Ryan, hey friends,
      we, Mobile Trooper a digital agency based in Germany, use ProcessWire for an Enterprise-grade Intranet publishing portal which is under heavy development for over 3 years now. Over the years not only the user base grew but also the platform in general. We introduced lots and lots of features thanks to ProcessWire's absurd flexibility. We came along many CMS (or CMFs for that matter) that don't even come close to ProcessWire. Closest we came across was Locomotive (Rails-based) and Pimcore (PHP based).
      So this is not your typical ProcessWire installation in terms of size.
      Currently we count:
      140 Templates (Some have 1 page, some have >6000 pages)
      313 Fields
      ~ 15k Users (For an intranet portal? That's heavy.)
      ~ 195 431 Pages (At least that's the current AUTOINCREMENT)
       
      I think we came to a point where ProcessWire isn't as scalable anymore as it used to be. Our latest research measured over 20 seconds of load time (the time PHP spent scambling the HTML together). That's unacceptable unfortunately. We've implemented common performance strategies like:
      We're running on fat machines (DB server has 32 gigs RAM, Prod Web server has 32gigs as well. Both are running on quadcores (xeons) hosted by Azure.
      We have load balancing in place, but still, a single server needs up to 20 sec to respond to a single request averaging at around about 12 sec.
      In our research we came across pages that sent over 1000 SQL queries with lots of JOINs. This is obviously needed because of PWs architecture (a field a table) but does this slow mySQL down much? For the start page we need to get somewhere around 60-80 pages, each page needs to be queried for ~12 fields to be displayed correctly, is this too much? There are many different fields involved like multiple Page-fields which hold tags, categories etc.
      We installed Profiler Pro but it does not seem to show us the real bottleneck, it just says that everything is kinda slow and sums up to the grand total we mentioned above.
      ProCache does not help us because every user is seeing something different, so we can cache some fragments but they usually measure at around 10ms. We can't spend time optimising if we can't expect an affordable benefit. Therefore we opted against ProCache and used our own module which generates these cache fragments lazily. 
      That speeds up the whole page rendering to ~7 sec, this is acceptable compared to 20sec but still ridiculously long.
      Our page consists of mainly dynamic parts changing every 2-5 minutes. It's different across multiple users based on their location, language and other preferences.
      We also have about 120 people working on the processwire backend the whole day concurrently.
       
      What do you guys think?
      Here are my questions, hopefully we can collect these in a wiki or something because I'm sure more and more people will hit that break sooner than they hoped they would:
       
      - Should we opt for optimising the database? Since >2k per request is a lot even for a mysql server, webserver cpu is basically idling at that time.
      - Do you think at this point it makes sense to use ProcessWire as a simple REST API?
      - In your experience, what fieldtypes are expensive? Page? RepeaterMatrix?
      - Ryan, what do you consider as the primary bottleneck of processwire?
      - Is the amount of fields too much? Would it be better if we would try to reuse fields as much as possible?
      - Is there an option to hook onto ProcessWires SQL builder? So we can write custom SQL for some selectors?
       
      Thanks and lots of wishes,
      Pascal from Mobile Trooper
       
       
    • By mr-fan
      What i wanna achive is a simple counter like that count up on visit (this is no problem) AND save the specific date (year/month/day) of the count...
      in the end i will be able to get visits per day/per month/per year in a nice and dirty graph.
      Just to have a way better simple counter system.
      Should i only go with a complex setup of pages like this:
      --stats (home template for pageviews)
      ----2018 (year)
      ------08 (month)
      ---------29 ->page_views   (integers on every day template)
      ---------30 ->page_views
      Or just simple use:
      --stats (home template for pageviews)
      ---->count (template) that holds simple field page_views and a date field
      or could a fieldtype like tables (one table field for every month/year or so) be also a solution?
      Or a own SQL table special for this and use it in a module? I don't have any experience on this topic...
      What i have in mind of performance sideeffects on such a thing?
      Or is there a solution that works with PW?
      I wanna go the hard way and implement something like this:
      http://stats.simplepublisher.com/
      only directly within PW and use the API to get the data...maybe create a simple module from it later i don't know if i  could set it up right from the start 😉
      this is the reason for my questions on more experienced devs
      Kind regards mr-fan
       
    • By FrancisChung
      Long but well written, detailed and informative article written by an Engineering Manager for Google Chrome about the true cost of Javascript and what you can do to alleviate some of that cost.
      Must read!

      https://medium.com/@addyosmani/the-cost-of-javascript-in-2018-7d8950fbb5d4
    • By dzervas
      Hello, I want to develop an eshop with processwire + padloper + any needed module (ex. ProCache n stuff).
      Right now, the eshop is in CS-Cart and I hate it. Will pw be able to support "proper" eshop features such as some simple sales graphs, 1k+ products, shipping methods & costs, user registration, search history, order status, payment status (for bank transfers for example) etc.
      For example I'm thinking that products and orders as sub-pages will be problematic (10k+ sub-pages)
      I also want to know if it's actually worth it or if it would be better to move to a bigger platform which has some of these features out of the box.
      Currently, the problem with CS-Cart is major lack of documentation which limits me in writing custom features (connection with shipping providers, export of data via CardDAV and more). Also CS-Cart is too complex to handle without documentation. I can manage and understand the code of PW but CS-Cart is just too big.

      Of course the server is beefy enough to handle things even heavier than CS-Cart. My focus is on development process and workflow (I've already set up CD from github to a docker container for a small website)
       
      EDIT: Sylius and Thelia2 really caught my eye. Comments on them will be appreciated
    • By pleini
      Hello,
      i have encountered a random problem with my processwire page. I have build a simple php file where i include the processwire index.php and run 1 simple database query and echo the total execution time of the script. I call this file periodically to check the performance of the page. Most of the time the execution time is under 0.5 seconds, but some times i goes up to over 3 or 4 seconds. The database query takes very little time, so the problem seems to be in the code called in index.php. 
      I checked my server and database metrics and they look perfectly fine. Has anybody experienced similar issues before? Or do you guys have any tips how to debug that issue the best?
       
      Thanks for your help in advance
      BR
      Joscha
×
×
  • Create New...