A Cautionary Tale, how the slightly lazy admin got burned.

Okay, so picture the scene, it’s Friday and I have a rule on Fridays, don’t change stuff. Friday is a day for getting the more mundane tasks out of the way, answer the emails I have ducked for a day or two, write some reports, tidy the mail box up, start on the personal development plan that I have been promising myself I was going to do for ages, and this time a really will, honestly…

This is how I want my Friday to be,

Not like this,

So got the phone call, the boss wanted to know why a virtual machine kept trying to migrate to another Host, ummm, I was thinking, aye, I keep an eye on every single system at all times so will know instantly what the hell you are talking about.

So I migrate all the VMs onto another host and all that’s left is the affected machine, but it’s the vCenter server, that goes down and I can’t centrally administer everything. So, tried a manual migration, didn’t work, the error comes up, but you know something, its Friday, and my eyes keep sloping off the screen and I think aw come on, don’t start this with me, not on a Friday.

So instead of hitting the forums and Knowledge Bases to figure out what’s going on I reboot the host.

It comes back up, I start the vCenter server up and boof, first problem ”That it isn't supposed to do that”, ah that old chestnut. The horrible feeling where you think, this is going to go very wrong.

You see all the hosts are on a distributed switch, but the distributed switch config is on vCenter server, so it can’t see the switch config and won’t talk to the network, ummmm, okay, starting to sweat a bit here

It’s fine though, I’ll just change the network mapping…oh that’s right I can’t, so what now.

Well in my attempt to be a lazy admin I have now made myself soooo much more work. So, off to the Host config

But of course, that’s no good, the Host still held on to the old distributed switch config. So idrac connect to the host and reset the network config, not a hard job, but setting it back up is when you don’t have all the networking info, like subnet masks and vLAN ID! Did I tell you that I hadn’t taken a note of them? So, head scratching and phoning the network guys ensues.

Then a moment of inspiration, check another host and get the details. Aye, it was Friday, I want firing on all cylinders. I go in, set the management IP and the vLAN, still no ping from the VM.

Till I went back in and had one of those moments…

Hadn’t enabled all the NICs, so the one with the relevant vLAN traffic wasn’t enabled, yup, a d’oh moment and a half.

Okay so back to the VM and the direct connection via the host, coz remember this a chicken egg situation, can’t admin the distributed switch because vCenter isn’t up and can’t admin vCenter coz I have no access to the distributed switch details. Removed the vNICs from where they were and added them to a Standard switch and boof! Back in business, well it was pingable and I could get back into the server and could then also start up vCenter via the vSphere client and get to my distributed switch again.

As always though, the VM waters can be shark infested

I had to remove the Host from the dSwitch (distributed switch), then blow away the standard switch at the same time as importing the host, the vNICs and assigning them to the proper VMK objects, and scratching my head as to why I had an extra connection that doesn’t seem to do anything. All the while running a continuous ping to make sure the damn thing stayed on the network.

I wish that the above was the case, but it works and I learned a lesson. Fix the original problem because sometimes the old favourite of have you restarted it sometimes just doesn’t work.

If this has helped one person avoid a near disaster then I am a happy man!

---

If you were entertained or somewhat diverted by this post please feel free to Upvote, Resteem and follow @wisbeech, you have been awesome for stopping by!

---

Images courtesy of Google
gif courtesy of giphy