CPU overload. [partly solved]

[Joss]

I went to update a site today, but as soon as I hit the backup databsase button, it shot to 98% CPU.  I tried it on another PW site and that did it when I hit the Upgrade link.

Any bright ideas what might be causing it?  PHP version?  (I am on 7.4 and made the change relatively recently.) Something with HTACCESS perhaps that was changed with the last update I did? (I think I updated the Pro-cache module not long back)

I tried turning on Debug, but since the site gets stuck, it doesn't cough up anything.

And hello to everyone!

 

Joss

 

EDIT

The solution or workaround is not to hit the backup button, but to just backup with phpmyadmin (which I do anyway).

However, why it is getting stuck, I am not sure.  This seems to be less about CPU overload and more about timing out.  See my later post with the reply from the host who has been brilliant.

 

[Joss]

As a tiny update, Kualo, my hosts, are playing with it. They replicate the problem every time, but the logs have yet to cough up anything useful. 

[Joss]

The host has been playing and they keep seeing this in the logs.  At this point, I am out of my depth!

Error: Exception: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
In /wire/core/WireDatabasePDO.php line 499
1. /wire/core/WireDatabasePDO.php(499): PDOStatement->execute()
2. /wire/core/PagesLoader.php(764): WireDatabasePDO->execute(Object(PDOStatement))
3. /wire/core/Pages.php(706): PagesLoader->getById(Array, Object(Template), 29)
4. /wire/core/PagesType.php(388): Pages->getById(Array, Array)
5. /wire/core/Users.php(58): PagesType->get(2138)
6. /wire/core/Session.php(180): Users->get(2138)
7. /wire/core/ProcessWire.php(520): Session->__construct(Objec
 

They tried increasing the resources to see if that did anything, but no. It is still failing.

One note - the MySQL server is there. It hasn't gone anywhere!

Joss

[horst]

Which was the previous PW version and to which version do you want(ed) to upgrade?

Are you using an upgrade module? If yes, is it up to date?

Can you simply copy the latest stable wire folder via ftp or ssh to the server and disable the upgrade module with a dot  in front of the directory name (if this resides in the site profile, and not under wire).

 

[Joss]

Hi Horst.

Trying to upgrade to the current dev. 

Although I could do a manual upgrade, that doesn't solve the problem as to why this is going wrong in the first place. I have tried on two different installations which I think are on slightly different PW versions.

The upgrade module is up to date on one and not on the other. So I don't think it is that as such.  There is a conflict somewhere, and it is recent. I upgraded a couple of other installations on the same server a couple of weeks ago without a problem.

[Robin S]

23 hours ago, Joss said:

as soon as I hit the backup databsase button, it shot to 98% CPU

Sounds like a problem with backing up the database rather than the version you are trying to update to. Are you able to do a manual database backup (via ProcessDatabaseBackups) or do you get the same CPU issue?

I recently had an issue with database backups but it was causing a memory issue rather than CPU. Turned out it was due to having SessionHandlerDB installed and the sessions were never getting deleted, resulting in a massive sessions table as described in this topic (along with the fix):

Unless you have SessionHandlerDB installed this won't be the problem in your case, but it would still be good to check your database in PhpMyAdmin or Adminer to see if there are any very large tables that could be taxing the CPU during a backup.

[Joss]

HI Robin

I can back up with Phpmyadmin okay.  I am not sure what SessionHandlerDB is, unfortunately! There wont be any large tables - the site is not that big. Also, I have updated this site before without any issues.

What about the list of errors the host sent to me? (up a couple of posts).  Does any of that give a clue?

[wbmnfktr]

3 hours ago, Joss said:

Error: Exception: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
In /wire/core/WireDatabasePDO.php line 499
1. /wire/core/WireDatabasePDO.php(499): PDOStatement->execute()
2. /wire/core/PagesLoader.php(764): WireDatabasePDO->execute(Object(PDOStatement))
3. /wire/core/Pages.php(706): PagesLoader->getById(Array, Object(Template), 29)
4. /wire/core/PagesType.php(388): Pages->getById(Array, Array)
5. /wire/core/Users.php(58): PagesType->get(2138)
6. /wire/core/Session.php(180): Users->get(2138)
7. /wire/core/ProcessWire.php(520): Session->__construct(Objec

I had very very similar entries in my logs on not that cheap hosting providers over in Switzerland and here in Germany ... I can't remember all the details but the end-result was... their database servers where more crowded than any Walmart on a Black Friday. Those servers were packed with users that even minimal resources couldn't be applied to each and every client.

I hesitate to say your hosting provider is [not that good] but for me it looks the database server is lower spec than a Raspberry Zero. Or at least connections to it, memory and maybe even disc space or whatever is insufficient.

There is a very huge company (EIG) out there, that buys small, medium and large hosting companies just in order to milk them. Similar to Hostgator and others. Maybe that's the case here too.

https://www.shoutmeloud.com/hostgator-sold-endurance-endurance-group.html

[adrian]

How big is the DB for the site? The backup mechanism built into the upgrade process (and also via ProcessDatabaseBackups) doesn't handle large DB's very well at all. I actually recently requested that Ryan consider changing the latter to use native backup tools (ie mysqldump) to avoid this.

Does the upgrade work OK if you just tell it not to backup? I don't honestly see the point in the backup while upgrading - PW upgrades almost never make changes to the DB anyway.

[Joss]

9 hours ago, wbmnfktr said:

I had very very similar entries in my logs on not that cheap hosting providers over in Switzerland and here in Germany ... I can't remember all the details but the end-result was... their database servers where more crowded than any Walmart on a Black Friday. Those servers were packed with users that even minimal resources couldn't be applied to each and every client.

Thanks, wbmnfktr. I use Kualo and they are very robust. I have used them for years.  They are being very helpful trying to work out what is happening.  The resource use on the particular server is quite low. 

8 hours ago, adrian said:

How big is the DB for the site? The backup mechanism built into the upgrade process (and also via ProcessDatabaseBackups) doesn't handle large DB's very well at all. I actually recently requested that Ryan consider changing the latter to use native backup tools (ie mysqldump) to avoid this.

Does the upgrade work OK if you just tell it not to backup? I don't honestly see the point in the backup while upgrading - PW upgrades almost never make changes to the DB anyway.

Hi, Adrian.  I am going to try upgrading without hitting the backup button on a different site (I want to leave the site we are messing with as is till I work out what has gone wrong).  But the DB is very small. Just 61mb.

[Joss]

I have had an email from support at Kualo, and I must give them a positive shoutout. They really have gone into this in detail. Brilliant.  Anyway, this is what they have said (which I have slightly redacted)

Quote

Hi Joss,

Thanks for the patience,

We have further analyzed the case with the team. Debugging the process carefully, the error occurs after 1 minute 40 seconds. Viewing the Timeout values of the database is set by default to 2 minutes, which should be enough.
Checking the base process we see that in fact it is not performing any action and remains in sleep:

| 2316866 | PASSWORD | localhost | PASSWORD| Sleep | 78 |

The execution that delays and cuts the connection is a PHP process:

ps -aux | grep XXX
XXX 3129 94.3 0.0 547792 104252 ? Rs 20:56 0:02 lsphp:ngb/public_html/ABCDEFG.com/index.php

That by making a trace we see that it does the following:

strace -p 3129
rite(10, "\nINSERT INTO `process_jumplinks_"..., 256) = 256
write(10, "\nINSERT INTO `process_jumplinks_"..., 268) = 268
write(10, "\nINSERT INTO `process_jumplinks_"..., 239) = 239
write(10, "\nINSERT INTO `process_jumplinks_"..., 229) = 229
write(10, "\nINSERT INTO `process_jumplinks_"..., 232) = 232

Pushes data into a thread, possibly the database dump.

By way of tests, we have applied an unlimited boost to the account and raised the execution times to more than 20 minutes and even then the process fails. Investigating a bit the error specifically of Processwire I found a quite old thread but very similar to the current function:

https://processwire.com/talk/topic/2580-mysql-server-has-gone-away/

Being the current parameters of the resources, we could estimate that it is a kind of bug in the execution of the backup function of the processwire.
In this case, I think that the post you made in the forum is the most appropriate to determine if it is a bug or if there is an appropriate fix.

Does that throw any light on the situation for anyone?

[Joss]

8 hours ago, adrian said:

Does the upgrade work OK if you just tell it not to backup? I don't honestly see the point in the backup while upgrading - PW upgrades almost never make changes to the DB anyway.

Adrian, I have confirmed it works without hitting the backup button. ? 

[cb2004]

Do you have the 404 monitor switched on in Jumplinks?

[Joss]

1 hour ago, cb2004 said:

Do you have the 404 monitor switched on in Jumplinks?

I think I do.  Why?

[teppo]

3 hours ago, Joss said:

I think I do.  Why?

It can result in a lot of data being stored in the database. From your earlier post it sounds like that could be at least one reason why your database takes so long to back up.

In my experience a ProcessWire database taking very long to migrate/backup/etc. usually means that the site has at least one, and possibly more than one, module that keeps storing data without limits (or with very loose limits). Jumplinks' 404 monitor is one module/feature that can do this, while ProcessChangelog and ProcessLoginHistory are other typical culprits.

I believe Jumplinks has an admin tool for checking the number of stored 404 hits and clearing them, so it might be a good idea to check that, just in case ?

[Joss]

26 minutes ago, teppo said:

It can result in a lot of data being stored in the database. From your earlier post it sounds like that could be at least one reason why your database takes so long to back up.

In my experience a ProcessWire database taking very long to migrate/backup/etc. usually means that the site has at least one, and possibly more than one, module that keeps storing data without limits (or with very loose limits). Jumplinks' 404 monitor is one module/feature that can do this, while ProcessChangelog and ProcessLoginHistory are other typical culprits.

I believe Jumplinks has an admin tool for checking the number of stored 404 hits and clearing them, so it might be a good idea to check that, just in case ?

Hi Teppo!

I turned it off and cleared it. I notice in the notes it says that it is limited to the last 100 only. I don't know if that means it only displays the last 100 or only keeps the last 100.

But the entire database is not very big.  5000 rows total? A couple of megabytes? It is still a bit of a puzzle. Kualo couldn't see an obvious reason. I also got this with another small site that doesn't have jumplinks installed.

HOwever, I think the logs have gone nuts. I am getting this error about not finding the Hanna Code text formatter.  I am just updating Wire because of another error (not displaying fields or templates in setup). 

[Joss]

This is the error I am getting:
Fatal Error: Uncaught Error: Class 'TextformatterHannaCode' not found in site/modules/HannaCodeHelper/HannaCodeHelper.module:111

and

Fatal Error: Uncaught Error: Call to undefined function wire() in /site/assets/cache/HannaCode/inline-ad.php:5 Stack trace:

 

[Joss]

This is my hanna code

<?php
			if($ad == "random"){
			$products=wire("pages")->get("/settings/adverts/")->children;
			$product=$products->getRandom();
			}else{
			$product=wire("pages")->get("/settings/adverts/$ad");
			}
			if($product){
			echo "<div class='footad'>";
		    		
		    		if($product->amazon_paste){
		    		echo $product->amazon_paste;
		    		}
		    		
		    echo "</div>";
		}

 

[Joss]

@teppo

Okay, I have just uninstalled Hanna Code Helper which appears to have been generating a lot of log entries! One every few seconds. It appears to have started 4 days ago, about the same time as I had problems backing up.

[adrian]

Did you recently update the main Hanna module? @teppo's helper hasn't been updated to support that new version yet. 

[Joss]

Just now, adrian said:

Did you recently update the main Hanna module? @teppo's helper hasn't been updated to support that new version yet. 

Yep! And I think I did it just before I had all these backup woes.

[gmclelland]

@Joss I had the same error https://github.com/ryancramerdesign/ProcessHannaCode/issues/23