MariaDB InnoDB crashing?

[Jonathan Lahijani]

(I'm putting this in the Dev Talk forum since I don't think this is ProcessWire specific.)

I have a ProcessWire site on a DigitalOcean server using Ubuntu 24.04 with MariaDB and PHP.

This site doesn't receive much traffic.  On 4/1/2025 MariaDB crashed.  Here's what systemctl said:

> systemctl status mariadb.service

× mariadb.service - MariaDB 10.11.11 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2025-04-02 06:31:12 UTC; 9s ago
   Duration: 1month 5d 13h 27min 44.368s
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 752113 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 752115 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 752118 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set->
    Process: 752178 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 752178 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 178ms

Apr 02 06:31:12 myserver1 mariadbd[752178]: 2025-04-02  6:31:12 0 [ERROR] InnoDB: Failed to read log at 73992704: I/O error
Apr 02 06:31:12 myserver1 mariadbd[752178]: 2025-04-02  6:31:12 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
Apr 02 06:31:12 myserver1 mariadbd[752178]: 2025-04-02  6:31:12 0 [Note] InnoDB: Starting shutdown...
Apr 02 06:31:12 myserver1 mariadbd[752178]: 2025-04-02  6:31:12 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Apr 02 06:31:12 myserver1 mariadbd[752178]: 2025-04-02  6:31:12 0 [Note] Plugin 'FEEDBACK' is disabled.
Apr 02 06:31:12 myserver1 mariadbd[752178]: 2025-04-02  6:31:12 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Apr 02 06:31:12 myserver1 mariadbd[752178]: 2025-04-02  6:31:12 0 [ERROR] Aborting
Apr 02 06:31:12 myserver1 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Apr 02 06:31:12 myserver1 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Apr 02 06:31:12 myserver1 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.11 database server.

Not sure why that happened and this is a recently built ProcessWire site (no legacy cruft).  I thought simply restarting MariaDB would fix it but it didn't.  Not even restarting the server fixed it.

After Googling and ChatGPTing, this post (and ChatGPT) recommended removing /var/lib/mysql/ib_logfile0 (and iblogfile1), then restarting.  The replies to that post suggest that while it works, it's dangerous for reasons related to potentially losing data.  Given that this site is not mission critical, that wasn't a concern and the approach suggested worked.  Ok.

---

Now today, on a totally different site and server (still DigitalOcean) with the same Ubuntu version and LAMP stack, MariaDB crashed in the same way.  I ran the same command "systemctl status mariadb.service" on that server and the output was basically identical.

Restarting MariaDB worked in this case so it was much easier to fix.

Again, this site doesn't receive much traffic.

---

I'm wondering what's going on here?  While these sites do get the typical hack-bots trying to exploit WordPress or insecure env files and things like that, I don't think that's the story here, even though they do tend to hammer the server.  It has to be one of the following reasons logically speaking:

• a bug with MariaDB and/or Ubuntu?
• a bug with ProcessWire?
• an issue with DigitalOcean underlying hardware? Unlikely.
• overloading of server resources from hackbots or even AI crawlers?  I looked at my logs and that didn't seem immediately plausible
• something code related on my end?  Possible because they are both running my home-grown ecommerce module.

Anyone else have issues with MySQL or MariaDB crashing for odd reasons?