Server overload help

[benbyf]

HELLO!

I've had two Timeouts recently on a site due to the server being overloaded.

The site is hosted on digitalocean on a single distro with 512 MB Memory, 20 GB disk, Ubuntu 14.04.4 x64. as shown below before each outage I can a climbing peak in user bandwidth then the CPU hangs:

(the light blue above is user, and the dark blue sys)

Not really had anything like this before with processwire as am puzzled. the site gets between 200 - 1000 visits a day, and I'm using the PW built in caching.

Any advice or tips would help loads!

[benbyf]

found this in logs:

exceptions: errors

DirectoryIterator::__construct(/srv/users/serverpilot/apps/betaville/public/site/assets/cache/Page/1/): failed to open dir: No such file or directory (in /wire/core/CacheFile.php line 208)     Error: Exception: DirectoryIterator::__construct(/srv/users/serverpilot/apps/betaville/public/site/assets/cache/Page/1/): failed to open dir: No such file or directory (in /srv/users/serverpilot/apps/betaville/public/wire/core/CacheFile.php line 208)

[adrian]

Do you think that log entry is related?

Have you checked the value of $this->path when this is called (line 208 of CacheFile.php)?

$dir = new DirectoryIterator($this->path);

 

[benbyf]

not sure how to do that without breaking my site (which i did abit just now)

[adrian]

1 minute ago, benbyf said:

not sure how to do that without breaking my site (which i did abit just now)

You might be able to do a: bd($this->path); before that line (if you have Tracy installed). Depending on when the CacheFile.php is loaded, this may or may not work. Turning on Tracy's Dump Recorder and Preserve Dumps option may also help.

Alternatively you could do: wire('log')->save('testing', $this->path); and then check the value recorded in the 'testing' log file.

[benbyf]

doing that I get a constant flow of paths (denoting someone accessed), but not all corrispond with an error, so it must be caching some and not others...

[benbyf]

Not sure if this makes a difference, but lots of pages have caching enabled but only two have folders in the assests/cache/Page/ home (1) and rss-feed (1029).

[DaveP]

Guessing, but this sounds like a permissions issue. /cache/ & children writeable?

[teppo]

Am I reading your graphs correctly, i.e. is there a notable increase in inbound network activity when this happens? If so, is that natural traffic (regular visitors browsing the site) or something else?

[Sergio]

Maybe a silly question, but is Swap enable and is it at least 2GB?

[Sergio]

44 minutes ago, teppo said:

Am I reading your graphs correctly, i.e. is there a notable increase in inbound network activity when this happens? If so, is that natural traffic (regular visitors browsing the site) or something else?

 

Good point. Check for spiders on the log.

[benbyf]

44 minutes ago, teppo said:

Am I reading your graphs correctly, i.e. is there a notable increase in inbound network activity when this happens? If so, is that natural traffic (regular visitors browsing the site) or something else?

should be normal vistors, but not sure how i would tell...

1 minute ago, Sérgio said:

Maybe a silly question, but is Swap enable and it's at least 2GB?

not sure what you mean here...

 

2 hours ago, DaveP said:

Guessing, but this sounds like a permissions issue. /cache/ & children writeable?

thought so too but its currently 775 on a user and im not going to go 777 for obivous reasons.... but why would it not have a problem caching the other pages, or not...? thats the strange thing, im only ever seeing those two pages id's in teh cache folder and in errors

[benbyf]

3 minutes ago, Sérgio said:

Good point. Check for spiders on the log.

might be an issue as there are LOOOOOOTSSSSSSS of pages

[Sergio]

9 minutes ago, benbyf said:

not sure what you mean here...

Disk swap is needed when some portion of disk space must be reserved to be used when the system RAM is full, so the disk acts like RAM (although much slower).

The default is zero on a fresh Ubuntu install and for lower RAM servers like this one should be at least the double (it's recommended 2GB for 512MB).

Serverpilot (that you're using) enables 512MB only: https://serverpilot.io/community/articles/swap-is-enabled.html

So I recommend you to log as root and increase it. 

https://www.digitalocean.com/community/questions/how-to-change-swap-size-on-ubuntu-14-04

And look here too: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04

[Sergio]

6 minutes ago, benbyf said:

might be an issue as there are LOOOOOOTSSSSSSS of pages

As you're using Nginx in front of Apache, maybe it's worth to try this: https://github.com/mariusv/nginx-badbot-blocker

[benbyf]

Will check and get back thanks!

[teppo]

12 hours ago, benbyf said:

should be normal vistors, but not sure how i would tell...

First thing to check is whether those requests look natural, even if just to rule out the off chance of a DDoS attack and stuff like that. Tons of requests to a specific page and usually something uncacheable (like search page with the same search term every time) is a telltale sign that someone is intentionally bombarding your site.

It's also good to check where those requests are pointed at: if there's a lot of traffic to a specific part of the site at the same time when issues start, the problem might be related to that part too. Or it could just mean that a specific piece of content is extremely popular

Most properly configured and resourced servers should be able to handle bot traffic with ease. There's the possibility of a misconfiguration and the settings of Apache, MySQL, PHP, etc. can always be tweaked in order to handle traffic spikes better, but if the root cause is that you're being harassed, there are other (and more efficient) ways to handle that.

[benbyf]

14 hours ago, Sérgio said:

As you're using Nginx in front of Apache, maybe it's worth to try this: https://github.com/mariusv/nginx-badbot-blocker

sweet! added, so hopefully that'll help

14 hours ago, Sérgio said:

Disk swap is needed when some portion of disk space must be reserved to be used when the system RAM is full, so the disk acts like RAM (although much slower).

The default is zero on a fresh Ubuntu install and for lower RAM servers like this one should be at least the double (it's recommended 2GB for 512MB).

Serverpilot (that you're using) enables 512MB only: https://serverpilot.io/community/articles/swap-is-enabled.html

So I recommend you to log as root and increase it. 

https://www.digitalocean.com/community/questions/how-to-change-swap-size-on-ubuntu-14-04

And look here too: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04

Also good to know! not encountered swapfile before and its also to know.

 

I feel like that might be something else going on though. I'm running PW 2.7.2 on this site and I opened up permissions to 777 briefly on assets/cache/Page and still got the errors in the logs, e.g.:

Error: Exception: DirectoryIterator::__construct(/srv/users/serverpilot/apps/betaville/public/site/assets/cache/Page/1/): failed to open dir: No such file or directory (in /srv/users/serverpilot/apps/betaville/public/wire/core/CacheFile.php line 208)

 

[Sergio]

Check the owner and permissions on site/assets/*

And if you want to learn more server stuff, there are a book and a video tutorial series on https://serversforhackers.com/ 

[jacmaes]

Speaking of bad bot blockers, I've had great success with Jeff Starr's robust "6G Firewall" for Apache:

https://perishablepress.com/6g/

[benbyf]

Added ProCache today which has helped alot on my server stats, as the CPU has gone from around 50-60% right down to 20%. however my RAM usage is still up the 80% which isnt great. Analytics are still at around 300-1500 hits a day. Will check out that site @Sérgio for sure.