Awful load times after upgrading to ProcessWire 3

[Marc]

I have upgraded a pretty large ProcessWire 2.7.2 website to version 3.0.25, and the site became much slower. On simple pages load times on average went from 555 ms to 780 ms, which is bad enough, but the more complex pages which gather info from other pages and add some custom queries went from 2.3 seconds to 5 seconds, twice as slow. A few facts:

• This is without template caching, which I cannot use on these particular pages.
• This is on localhost. I have not upgraded the live site yet (fortunately).
• Load times are on the slow side regardless because my local database server is on a NAS.
• There are no errors.
• By upgrading to PW 3 I mean I did the standard upgrade path by replacing the wire folder, overwriting index and htaccess files and letting the admin upgrade the db. I also added the ProcessWire namespace to all my custom template files and scripts.
• I'm not using any PW3-specific config options or features yet.
• I use very few plugins.

I'll be needing version 3 features for this particular website soon so I will have to do the upgrade either way. But I'm not sure how to troubleshoot these poor load times. Where do I start? Any pointers would be greatly appreciated!

Edit: I did some test with simple pages that use template cache: load times went up from 287 ms to 470 ms on avarage. I should also add these times are for the document to load, before scripts like CSS and images are loaded. So I'm measuring only the time it takes for ProcessWire to do its thing.

[flydev]

In before complete answer, the first thing I see is :

12 minutes ago, Marc said:

By upgrading to PW 3 I mean I did the standard upgrade path by overwriting the wire folder, index and htaccess files

What the doc say:

When you put in the new /wire/ directory, make sure that you remove or rename the old one first. If you just copy or FTP changed files into the existing /wire/ directory, you will end up with both old and new files, which will cause an error.

[Marc]

4 minutes ago, flydev said:

In before complete answer, the first thing I see is :

What the doc say:

When you put in the new /wire/ directory, make sure that you remove or rename the old one first. If you just copy or FTP changed files into the existing /wire/ directory, you will end up with both old and new files, which will cause an error.

Sorry I worded that incorrectly, I renamed the old wire folder first so I did not overwrite it. I'll change that sentence. 

[Marc]

After testing vanilla installations of the intermediate profiles of both PW 2.7.3 and PW 3.0.42, it seems PW 3 is just slower in general than PW 2. I'm seeing differences in load times of 30+ percent on identical pages. I suppose that's a price to pay for all the new stuff that is possible with version 3. The upside is, maybe this means the longer load times of the website I upgraded are to be expected. 

[kongondo]

11 minutes ago, Marc said:

After testing vanilla installations of the intermediate profiles of both PW 2.7.3 and PW 3.0.42, it seems PW 3 is just slower in general than PW 2. I'm seeing differences in load times of 30+ percent on identical pages. 

Hmm. That's an interesting observation. I am curios that no one else (as far as I can tell) has had/reported a similar experience. I wonder if it has to do with your setup? 

 

[Marc]

 

2 minutes ago, kongondo said:

Hmm. That's an interesting observation. I am curios that no one else (as far as I can tell) has had/reported a similar experience. I wonder if it has to do with your setup? 

 

I agree, that is suspicious and maybe I am jumping to conclusions. However in trying to rule out my local setup I uploaded my upgraded website to my server and saw the exact same degradation in speed. Which disappointed me greatly because I was really hoping it was just my local XAMPP setup causing this.

[Soma]

I can confirm this even without testing. I noticed that ProcessWIre is getting slower and slower each iteration. Even 2.73 was slower than all the previous. But seeing what all gets added to PW of course the overhead grows.

Also PW is very heavy on the server side and especially DB side and it can get somewhat slow really fast and produces tons of queries. We have very big and complex sites that suffer from some sort of performance issues due to so many table joins and not well indexed tables (said our hoster). We also are currently letting them investigate and may have some details soon.

I just did a quick install of 2.73 and 3 master, and can confirm the ~30% slower.

[Craig]

I recently experienced a similar effect too, across three sites on two very similar servers.

Running 5 requests with no caching against the PW Admin login page, as guest, these results are the median response time for just the page request:

• Server 1, PHP 5.6.15, Site A, PW 2.7.3: 217ms
• Server 2, PHP 5.6.22, Site B, PW 2.7.1: 293ms
• Server 2, PHP 5.6.22, Site C, PW 3.0.17: 672ms

I noticed the slower time when Site C was first launched, but had little time to really investigate it there and then, and haven't got round to it since.

[Pixrael]

I have also experienced this slowness for a while. Solving this issue should be at the top of the Roadmap for 2017 because of the effect it has on the SERPs and the User Experience. I think it's more important than new features or the expansion of existing ones. Today Processwire is so powerful and versatile, now should make a stop and solve this delicate problem before continuing to advance.

What do you think guys ? Ryan?

[tpr]

Do you get these values on the same server (same PHP version)?

[Soma]

Yep same server.

[BitPoet]

I concur that some kind of slow down is to be expected when complexity rises. Before touching template cache, markup cache or thinking about APC or Varnish though, the first step should IMHO always be to cache the raw PHP by using opcache or wincache (the latter should only be enabled after installing PW).

Having said that, I can see a few specific things that might impact performance in 3.x:

• Namespaces. Yes, they don't impact hard, but any call to a global function in namespace context makes PHP look for an identically named function in the current namespace. It might be worth it to go through the code and qualify all global calls, like e.g.  \explode() instead of explode(). I'll see if I can whip up a tool to do that.
• File system access / file compiler. Not much to do about directly it if you need it, besides faster disks.
• Memory footprint. More functionality, more properties, more memory, but should only be a real issue on slow or overloaded servers.
• Old PHP versions. Some PHP features needed a few release steps to reach optimal performance. It's always worth it to compare performance under e.g. 5.5 and 7.1.
• MySQL. Often, production systems using a distribution's default installation have performance schema enabled unnecessarily or are logging every tiny query to disk. Though this shouldn't impact in whole percentages unless the system's at its limits anyway.

[Klenkes]

2 hours ago, BitPoet said:

Old PHP versions. Some PHP features needed a few release steps to reach optimal performance. It's always worth it to compare performance under e.g. 5.5 and 7.1.

I absolutely agree!

I tested a testsite of mine with PHP 5.4(and 5.6) and noted the times for a few different pages with Tracy. Times like 750ms to 1.2 sec.

Then I switched to PHP 7 and generating times went down like crazy. Well... about 30-40% down. 500ms instead of 750ms and 400ms instead of 680ms

[ryan]

A stock PW3 installation should actually be faster than PW2 – it certainly is on all of my installations. There's a lot more optimization in PW3 than in PW2. If that's not what you are seeing, then it's time to start looking for where the bottleneck is. PW3 does not have more significant overhead than PW2 except when it is compiling a file for the first time. The 30% increase numbers mentioned above sound to me like that is a request where PW is compiling a file. You can expect a request where it has to re-compile a file to take longer. But it only has to do that once, when a file changes and needs to be re-compiled. Maybe that's a common occurrence on a dev site, but should be a rare one on a production site. I'm measuring here with Chrome dev tools, ProfilerPro and my own timers using the Debug class. What tool are you guys using to measure times, and in what quantity? Before deciding something is PW3 related, I really suggest testing with a basic/blank profile without other modules installed. If you are consistently seeing any kind of increase in render times under PW3, my guess would be that something is getting recompiled on every request for some reason or another, or that there is another module involved that runs slower under PW3 for some reason. 

Edit--A few things to add: 

Debug mode is going to be slower in PW3 than in PW2, simply because PW3 is tracking a lot more stuff than PW2 did. With debug mode off there should be no difference though. Keep in mind debug mode is for development, and not something you should ever leave enabled on a production site. 

PW3 is more efficient with resources than PW2. PW3's boot time (stuff it does before executing your template file) is 20% to 45% faster in my testing. PW3 executes 20% to 25% fewer queries as part of the boot, and loads up to 50% fewer pages. Autoload modules become part of the boot process, so I test with no 3rd party modules installed. 

The file compiler can potentially add a little bit of overhead even when it doesn't need to compile, because it has to determine whether something needs compilation. But on a default site profile we're talking about maybe 10ms at the most here. If you turn off the template compiler, then that overhead is gone. 

While PW3 uses fewer resources on the database side, but sometimes more resources on the file system side. If you've got a slow file system, you might notice it more in PW3 than in PW2. 

For those of you seeing PW3 to be slower than PW2, if it's determined that 3rd party modules are not a factor, I would be curious what's happening in the template files. Perhaps there is a bottleneck in a certain API call or something that we're not aware of yet. It would be interesting to see the results of profiling the API calls in your template files using ProfilerPro or Debug::timer() calls. 

Mostly specific to Soma: PW2 and PW3 are identical in terms of how they use joins and indexes. Regardless of version, PW will use as many joins as it takes to execute the selector you give it. Just like you can create complex or inefficient SQL queries you can also create equally inefficient $pages->find() calls if you aren't being careful. Using PW's API doesn't mean you are somehow bypassing the database. Your find() queries still become SQL queries. So if you are working on big and complex projects, then you need to watch and profile your work. When you a come across a complex find() operation that is expensive, refactor it to be simpler or break it down into smaller parts. Pay attention to how many pages you load in memory at once. Don't use find() and children() calls without "limit" selectors when dealing with potentially large sets. With regard to indexing, PW logically indexes all the columns that are likely to be used in find() operations, but if you are querying columns in a table have no index for whatever reason (3rd party module that forgot an index, or column not commonly used for queries), you may need to add one. Most of us never need to do this, but since you mentioned "big and complex" you may be in the territory where you have to apply more consideration to these things. 

[Marc]

Thank you Ryan for the detailed response. Given all of the optimizations you mention it is indeed quite strange myself and a few others are seeing this behavior.

22 hours ago, ryan said:

I'm measuring here with Chrome dev tools, ProfilerPro and my own timers using the Debug class. What tool are you guys using to measure times, and in what quantity?

I'm using the most basic way of testing which is Chrome dev tools and repeatedly refreshing the page to inspect the load times. Doing this with unmodified and brand new intermediate profile installations of Processwire 2.7.3 and 3.0.42 I see the 30% difference in load times. By unmodified I mean I haven't changed a single template or page that is installed by default. I only modified config.php to connect to my database, everything else is untouched (no extra modules). This is on a Windows machine with Xampp running PHP 7.0 on an SSD drive and a remote database connection (which is surely slowing things down but it should be slowing both the PW 2 and 3 database connections equally I imagine). 

I will setup a different development environment at home later today or tomorrow that is Vagrant/Ubuntu-based and repeat the test. To be honest I'm expecting no difference as I've already seen similar degradation on my production server where I uploaded a site that I upgraded to PW 3 (which has load time differences of up to 100%, ouch). 

I should note that adding $config->moduleCompile = false to the PW 3 site makes no difference.

[ryan]

Marc, it sounds like you've got a bottleneck somewhere. You mentioned a remote DB connection, and maybe there's something to to that. You also mentioned Windows (I'm in unix), and it's certainly possible there's something platform specific going on. Here are some things you can do to narrow in on where it might be. 

Edit your /index.php file and before the line that says this:

$config = ProcessWire::buildConfig($rootPath);

Add this:

Debug::timer('pwboot');

Now edit your /site/templates/_init.php file (or whatever gets called first) and add this at the top, after the "namespace ProcessWire;" (old school and rough, but gets straight to the point): 

die('Boot time: ' . (Debug::timer('pwboot')*1000) . ' ms');

Now you can get a sense of how long it takes ProcessWire to boot, before it hands off the request to you. This should give you an indicator as to whether you need to look at the modules you've got installed, or whether you need to look at the API calls in your templates. 

Hit reload several times in your site (front-end) to get a sense of the average boot time. On my oldish Macbook Pro and PHP 7, running the site-default profile with no 3rd party modules installed, these are my average boot times:

• ProcessWire 3.0.52: 53 ms (opcache off)
• ProcessWire 2.7.3: 104 ms (opcache off)
• ProcessWire 3.0.52: 35 ms (opcache on)
• ProcessWire 2.7.3: 35 ms (opcache on)

As you can see, without PHP caching the code, ProcessWire 3.x boots nearly twice as fast as ProcessWire 2.7 (at least in my environment). Interestingly, if PHP's cache is enabled, then the boot time is identical. But when Soma said above “ProcessWire is getting slower and slower each iteration”, this is a blanket statement that is the opposite of reality. ProcessWire is getting faster and more optimized with each version. What's real is that clearly something about the conditions present in his environment and your environment is reducing performance, and that needs to be narrowed in upon. So the next steps will be to determine if it's something about the server, something from a module, or something going on in the template file(s), like some specific API call that might be common among your installations. 

Now go to your Chrome dev tools, Network tab. Hit reload several times like you did before (same URL). Note the first request in your Network tab. Take this time and subtract the time you recorded above. This represents (roughly) the time required to load PHP before it gets to ProcessWire. With opcache off, my PHP takes about 100ms before it loads ProcessWire's /index.php file. If I enable opcache, then I'm seeing almost no overhead from PHP at all. Meaning, Chrome dev tools is showing 40ms or less. 

I'm guessing that your remote DB connection is going to mean your boot times are significantly higher than if it was local. So maybe you shouldn't be looking for the same boot times that I'm seeing here. But hopefully you can get a better idea of whether the times you are seeing are related to the boot process, or what happens after the boot process. If you are seeing it in the boot process, then uninstall all 3rd party modules and test again. Or go one-by-one until you can narrow in on which one it is. If still seeing a slow boot then we might need to start looking at potential Windows platform issues. If your boot times seem reasonable, then the next step is to start debugging in the template files. Let me know what you find and I can suggest the next steps. 

 

[Soma]

My test are also a clean installs 2.73 and 3.0.42 (default profile, no changes, except File Compiler disabled) locally on same Server/MYSQL/PHP without any cache. The difference is there around 30% slower measured in Chrome TTFB refreshing a couple times to get a average. Not sure how your test results are the other way around.

[Soma]

Testing this a little further. Stock Install, Debug off. Default Advanced Profile.

MAMP PHP 5.6.10 / OPC off

PW2 Boottime: ~120ms
PW3 Boottime: ~80ms

PW2 Rendertime Home: ~30ms (start of init, end of _main.php)
PW3 Rendertime Home: ~36ms (start of init, end of _main.php)

PW2 TTFB Chrome: ~170ms
PW3 TTFB Chrome: ~240ms

Not sure why the TTFB is so much higher.

[cstevensjr]

15 minutes ago, Soma said:

Testing this a little further. Stock Install, Debug off. Default Advanced Profile.

MAMP PHP 5.6.10 / OPC off

PW2 Boottime: ~120ms
PW3 Boottime: ~80ms

PW2 Rendertime Home: ~30ms (start of init, end of _main.php)
PW3 Rendertime Home: ~36ms (start of init, end of _main.php)

PW2 TTFB Chrome: ~170ms
PW3 TTFB Chrome: ~240ms

Not sure why the TTFB is so much higher.

Are you able to test with PHP 7.x?  If so, it would be interesting to see what the results are.  Thanks.

[Soma]

Yes I tested and it's about the same ratio except all a little lower.

Like 20ms - 26ms Rendertime, TTFB 130ms - 170ms

[Klenkes]

I did exactly the same test with exactly the same results as Soma, and I squeezed PW 2.8.35 in between, which resulted in values riight in between around 200ms with Chrome Dev Tools.

I even took a small website of a friend(PW 2.7.2) and copied it to 2 locations and updated one to 2.8.35 and the third to 3.0.42

At first the 3.0.42 was faster! About 30%. But after a lot of reloads to check the load times it got slower and slower and ended up last place.

This is weird...

[Marc]

Finally set up my Vagrant testing environment (provider Virtualbox) running a Ubuntu 14.04 server with PHP 7.1 with Nginx in front of Apache. I'll mimic Soma's report:

PW2.7.3   TTFB Chrome: ~28ms
PW3.0.42 TTFB Chrome: ~40ms

Those are new installs of intermediate profiles without modifications. I will look into the other numbers later.

[FrancisChung]

I've just upgraded to PW 3 (3.0.62) and I'm seeing minor speed improvements on our Test Server.

Before : 3.9s for full document load

After : 3.7s for full document load

Edit : I was on PW 2.8.x previously

[FrancisChung]

Update : Early days and take it with a slight pinch of salt, but we've seen some positive uptick in our behavioural stats.

Session duration up 5.3% vs previous month, Pages per session up 9.6%