What a night! We had a power outage at the office yesterday afternoon. The UPS wasn't able to keep things running for long enough, and so everything cam the a grinding halt.
The end result was that everyone went home, and I waited and watched from home until power returned to my server cabinets, hoping the automatic startup routine would swing into action, and I could stay home and do my checks from there. It didn't.
So off I went, back to the office at about 6pm to get things up and running again. 2 Of my switches didn't recover, and are now out of action, and there were a few other things wrong as well.
Fast forward a few hours, a database rebuild, a startup error on the storage module to fix, and my entire virtual environment on its knees, refusing to start up... and the phone system being completely unreasonable, it was going to be a long night.. Fortunately for me, I have a really good working relationship with our service provider, and one phone call to the director and help was on the way.
We got most things working just after midnight, but the phones were still down. At least staff coming in in the morning would still be able to work. Having 50 people doing nothing but ask you when it's going to be fixed would have driven me crazy, and cost the company a lot of miney for no work.
After 4 hours sleep I was back in the office, with my help, to get things working. We got it all up and running eventually, and with very little user disruption, which was great. It also forced me to tidy up the network cabling in the cabinet, to remove unused leads and relocate active ports from the dead switches onto the good ones.
Not a bad effort if I do say so myself.
Now I'm knackered and going home to relax. Job done!