CelticRadio.net was crippled this afternoon by a problem with our database. It is really just a case of traffic increasing to the site and to our database. The default settings are no longer good enough to keep things running smoothly.
Last week we increased the max connections to the database from 100 to 1000. Today I learned there is something called max_connect_error which is defaulted to 10 and when you reach 10 the database locks everybody out because it thinks it is a hacking attack. So we bumped that connect error to 2000 and increased the max connections to 2000 just to be on the safe side.
Ok, learning experience for me!
I'm going to think about a way members can contact us to notify of problems like this because sometimes we are not even aware. I do have my cell phone attached to [email protected], so if you selected "send a mobile" message it would be like a pager and I would receive a text message.
Long term I need to build some bells and whistles to go off when this sort of thing happens.