A Cautionary Tale, how the slightly lazy admin got burned.

in #technology7 years ago


Okay, so picture the scene, it’s Friday and I have a rule on Fridays, don’t change stuff. Friday is a day for getting the more mundane tasks out of the way, answer the emails I have ducked for a day or two, write some reports, tidy the mail box up, start on the personal development plan that I have been promising myself I was going to do for ages, and this time a really will, honestly…

This is how I want my Friday to be,

Not like this,

So got the phone call, the boss wanted to know why a virtual machine kept trying to migrate to another Host, ummm, I was thinking, aye, I keep an eye on every single system at all times so will know instantly what the hell you are talking about.

So I migrate all the VMs onto another host and all that’s left is the affected machine, but it’s the vCenter server, that goes down and I can’t centrally administer everything. So, tried a manual migration, didn’t work, the error comes up, but you know something, its Friday, and my eyes keep sloping off the screen and I think aw come on, don’t start this with me, not on a Friday.

So instead of hitting the forums and Knowledge Bases to figure out what’s going on I reboot the host.

It comes back up, I start the vCenter server up and boof, first problem ”That it isn't supposed to do that”, ah that old chestnut. The horrible feeling where you think, this is going to go very wrong.

You see all the hosts are on a distributed switch, but the distributed switch config is on vCenter server, so it can’t see the switch config and won’t talk to the network, ummmm, okay, starting to sweat a bit here

It’s fine though, I’ll just change the network mapping…oh that’s right I can’t, so what now.

Well in my attempt to be a lazy admin I have now made myself soooo much more work. So, off to the Host config

But of course, that’s no good, the Host still held on to the old distributed switch config. So idrac connect to the host and reset the network config, not a hard job, but setting it back up is when you don’t have all the networking info, like subnet masks and vLAN ID! Did I tell you that I hadn’t taken a note of them? So, head scratching and phoning the network guys ensues.

Then a moment of inspiration, check another host and get the details. Aye, it was Friday, I want firing on all cylinders. I go in, set the management IP and the vLAN, still no ping from the VM.

Till I went back in and had one of those moments…

Hadn’t enabled all the NICs, so the one with the relevant vLAN traffic wasn’t enabled, yup, a d’oh moment and a half.

Okay so back to the VM and the direct connection via the host, coz remember this a chicken egg situation, can’t admin the distributed switch because vCenter isn’t up and can’t admin vCenter coz I have no access to the distributed switch details. Removed the vNICs from where they were and added them to a Standard switch and boof! Back in business, well it was pingable and I could get back into the server and could then also start up vCenter via the vSphere client and get to my distributed switch again.

As always though, the VM waters can be shark infested

I had to remove the Host from the dSwitch (distributed switch), then blow away the standard switch at the same time as importing the host, the vNICs and assigning them to the proper VMK objects, and scratching my head as to why I had an extra connection that doesn’t seem to do anything. All the while running a continuous ping to make sure the damn thing stayed on the network.

I wish that the above was the case, but it works and I learned a lesson. Fix the original problem because sometimes the old favourite of have you restarted it sometimes just doesn’t work.

If this has helped one person avoid a near disaster then I am a happy man!


If you were entertained or somewhat diverted by this post please feel free to Upvote, Resteem and follow @wisbeech, you have been awesome for stopping by!


Images courtesy of Google
gif courtesy of giphy

Sort:  

I absolutely love that first meme!!

Heh heh cheers dude.

Brilliant writing, it read a bit like
A fast-paced detective story as one problem led to another. Resteemed :-)

@wisbeech
Nice Job!
Keep the good work up!
Thanks for sharing

Thank you very much, I will.

Congratulations! This post has been upvoted from the communal account, @minnowsupport, by Scottish01 from the Minnow Support Project. It's a witness project run by aggroed, ausbitbank, teamsteem, theprophet0, and someguy123. The goal is to help Steemit grow by supporting Minnows and creating a social network. Please find us in the Peace, Abundance, and Liberty Network (PALnet) Discord Channel. It's a completely public and open space to all members of the Steemit community who voluntarily choose to be there.

If you like what we're doing please upvote this comment so we can continue to build the community account that's supporting all members.

Coin Marketplace

STEEM 0.17
TRX 0.15
JST 0.028
BTC 62345.27
ETH 2427.57
USDT 1.00
SBD 2.49