Ethernet Spaghetti - A Late Night at the Office

in life •  2 months ago

What a night! We had a power outage at the office yesterday afternoon. The UPS wasn't able to keep things running for long enough, and so everything cam the a grinding halt.

The end result was that everyone went home, and I waited and watched from home until power returned to my server cabinets, hoping the automatic startup routine would swing into action, and I could stay home and do my checks from there. It didn't.

IMG_04092018_135932_(700_x_700_pixel).jpg

So off I went, back to the office at about 6pm to get things up and running again. 2 Of my switches didn't recover, and are now out of action, and there were a few other things wrong as well.

Fast forward a few hours, a database rebuild, a startup error on the storage module to fix, and my entire virtual environment on its knees, refusing to start up... and the phone system being completely unreasonable, it was going to be a long night.. Fortunately for me, I have a really good working relationship with our service provider, and one phone call to the director and help was on the way.

We got most things working just after midnight, but the phones were still down. At least staff coming in in the morning would still be able to work. Having 50 people doing nothing but ask you when it's going to be fixed would have driven me crazy, and cost the company a lot of miney for no work.

After 4 hours sleep I was back in the office, with my help, to get things working. We got it all up and running eventually, and with very little user disruption, which was great. It also forced me to tidy up the network cabling in the cabinet, to remove unused leads and relocate active ports from the dead switches onto the good ones.

Not a bad effort if I do say so myself.

IMG_04092018_135911_(220_x_700_pixel).jpg

Now I'm knackered and going home to relax. Job done!





Team Australia, Team South Africa, and The Alliance banners by @bearone

My Awesome Upboks by @ryivhnn

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Nice job man. It is the stuff nightmares are made of.

At my old work we used to go into "Lockdown" and literally barricade ourselves in. Grab some admin or support staff to man the phones and handle the PR side of it so the techos could work without being bombarded.

Sounds like you got lucky this time ;)

·

Thanks mate, I was quite disappointed at how quickly the UPS died. I may need to get something a bit bigger so I at least get a chance to run through a more graceful shutdown of the systems.

Nothing brings out the worst in people than when they can't browse their online shopping 😜. I did get lucky this time for sure.

Oh man! That's one tidy server rack! The next guy that has to deal with that has better and you a nice bottle of something!

·

Thanks @bengy. I'm still not 100% happy with some of the cabling, but would have to unplug a fair bit more to get it neater. A nice bottle of something does sound good after a night like last night.

·
·

Haha, it can never be perfect! Too many things going in different directions. I'm glad you understood what I wrote... it seems like the autocorrect took over my comment for a little bit there at the end...

Ohhh, man I would have been cussing up a storm, I hate that kind of shit. Good on you for managing it!

Thank you for your continued support of SteemSilverGold