From the trenches: heavy slowdown after Meltdown + Spectre patch on a DigitalOcean droplet

[Sergio]

Hello fellow PW devs!

This is a short story from the server management trenches.

These past couple of days trying to solve an unexpected problem: after DigitalOcean patched the droplets in NYC3 region last week, my client's droplet became almost useless and went down a couple of times. The droplet has 2GB RAM and was running Ubuntu 16.04 that was updated to kernel 4.4.0-116 after the patch. The server was provisioned using Forge (forge.laravel.com).

After sshing into it, and running "top" I've noticed the cause: "php-fpm7.1" processes (3-5 instances) were spiking the CPU to 100%. This was very odd, as the CPU usually kept around 33% most of the time.

The site uses ProCache and markupCache and was getting around 800-1000 visits/day last week. I checked everything on PW's side and nothing seemed out of place, so I went restarting PHP and Nginx but the problem continued.  I checked access logs and no suspicious activity shown up. I upgraded PHP to 7.2 to see if anything will changed but the problem continued.

My only guess after all that is that the droplet in question got screwed up somehow, because I didn't see any complaints on the web of other people getting the same problem on DO (But I confess that I did a quick Google search only).

So in the end I decided to create a new droplet, now with 2 CPU cores and kept the 2GB (1 extra core and $5 cheaper ). Reinstalled PW there and pointed the floating IP to this new server.

The installation went smooth but to one issue: error log started to show messages of MySQL showing "to many files" error when the users were searching. I've never encountered this message before, so after reading some StackOverflow posts, I changed mysql.services config file to remove its file limit (https://stackoverflow.com/a/36807137)

Everything is normal now, but I think I'll never discover what truly happened. Anyone else had this kind of problem with MySQL before?