Site Crash

[Jennifer S]

Good morning. I'm having a problem with my big PW site. It stopped responding less than an hour ago (except for the 500 Internal Error message). Here is what the error log says:

2014-03-17 13:07:50    ?    https://wwwudev.cac.washington.edu/?/    Error:     Exception: SQLSTATE[HY000]: General error: 1194 Table 'modules' is marked as crashed and should be repaired (in /nfs/bronfs/uwfs/info/wwwudev/world/uaa/advising/wire/core/Modules.php line 251)
2014-03-17 13:13:30    ?    https://wwwudev.cac.washington.edu/?/    Error:     Exception: SQLSTATE[HY000]: General error: 1194 Table 'fields' is marked as crashed and should be repaired (in /nfs/bronfs/uwfs/info/wwwudev/world/uaa/advising/wire/core/WireSaveableItems.php line 141)
2014-03-17 13:19:02    ?    https://wwwudev.cac.washington.edu/?/    Error:     Call to a member function has() on a non-object (line 36 of /nfs/bronfs/uwfs/info/wwwudev/world/uaa/advising/wire/core/Users.php)
 

I repaired the modules and fields table but now I'm getting the last error and I don't know what is happening. The site is on a shared server; I checked to make sure I have lots of disk space. I don't know what could have changed this morning; my next step is to ask our network technicians.

 

[adrian]

Hi Jennifer,

It sounds like the repair worked on the tables.

The member function error on $user->roles makes me think you have a session issue - maybe? Have you tried emptying your assets/cache folder?

Really not sure, but maybe worth a try.

Can you access the admin, or is the entire site down?

[Jennifer S]

Thanks for the suggestion. I just emptied the assets folder. No change. Yes, public site and admin site both are down.

[adrian]

Is there any chance that some of the files in the PW wire folder could have been deleted? You could try replacing the entire wire folder with a fresh copy - that might be a good place to start?

Sorry, hopefully someone else will have a better idea

[Jennifer S]

Adrian, it looks like somehow I lost 50K of data in my database between today and yesterday. Both my fields and modules tables were repaired successfully but they had lost all their data.

I dropped and then re-installed this weekend's backup of the database and now the site is back up and running. 

Now to try and figure out what happened...

Thanks again for helping...

[adrian]

No problem - I wasn't much help

Always disconcerting when a db spontaneously loses data - good luck with the sleuthing!

[ryan]

What version of PW? I can't say as though I've ever seen tables get corrupted like that. But since you've got corrupted tables, you might also have a corrupted file system. I think Adrian's suggestion to replace your /wire/ directory is probably a good one.

Have any new modules been recently installed?

In your /site/config.php enable debug mode (change the line that says $config->debug=false; to $config->debug=true). You should now get a more detailed error message. Paste in the entire error message here if possible.

You mentioned that you cleared out your "assets" folder. I'm hoping that you cleared out your "cache" folder in assets, and not the entire assets folder? Still it shouldn't be necessary to clear out the cache folder.

Most likely something is still corrupted in the database or on the file system. You may want to export your entire database and re-import to a new database, then update it to point to that new database in your /site/config.php file (at the bottom). But while you are in PhpMyAdmin can you look at the list of tables and confirm that you have a "field_roles", and a "roles" table, both populated (not empty)? The error message you are getting makes me wonder if something is missing there. 

[ryan]

Looks like more replies came in while I was writing the above message. You mentioned you are on a shared server–who is it shared with? You may want to double check that your directories have safe permissions for a shared environment (if will vary per web host, and the web host can advise on what is best). Though since your tables needed repair, I would suspect something broke on the server (HD crash or corruption) rather than someone messing with the site. Have any new modules or code been recently added to the site?

[Joss]

I had a similar problem about a year ago on a non-pw site. It turned out that MySQL had crashed/corrupted on a shared solution and the hosting company had rebuilt it but something had gone wrong and not all the tables were complete.

Took me two weeks to get them to admit that the problem had been their end. (Which is slow for me .....)

[Jennifer S]

Good evening, folks. Well, about an hour after I reported my data loss to the server admins this morning, they sent out a message campus-wide saying there was a critical need to reboot the three mysql servers that run most of the U's websites.

I think we can safely ascribe this to a case of something being broken on the server, as Ryan suggests, although I may never find out what, exactly. When I looked at my tables more closely I saw that the "fields" table had zero entries, which sounds pretty sub-optimal. I'm glad I had a backup.

Joss, two weeks sounds like a long time to wait for clarity. I admire your persistence. It's hard not to be in control of the server machines. A bit like the electric company I guess.

Thanks, people, for thread-sitting this with me.

[onjegolders]

Tacking my question onto this thread as it has some similarities. I think our VPS provider has been having issues today with their servers and when I tried going to one of our sites we were getting "Unable to complete this request errors". When I turned on debugging, I get the following error

Fatal error: Exception: SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql_daf_0.MYI' (Errcode: 30) 

Which sounds to me like it's a problem accessing/writing to the database (on our server)? Could anyone confirm if that sounds right?

Thanks.