A couple of years ago, I wrote two short books for Microsoft Press called Optimizing and Troubleshooting Hyper-V Networking and Optimizing and Troubleshooting Hyper-V Storage. Both these books were written in collaboration with experts from the Windows Server team at Microsoft and consisted of short real-world scenarios of various issues involving Hyper-V hosts and virtual machines with explanations of how the problems were identified and resolved. The books were not meant as exhaustive technical guides for Hyper-V troubleshooting but merely to indicate the general approach and steps involved in fixing different issues customers of Microsoft had experienced.
Since the time of writing these two books I’ve continued to take an interest in Hyper-V troubleshooting, though more from a distance than actually being involved hands-on with such issues. This present article and some possible future ones I may write are intended to build upon my earlier publications with more examples taken from real life of problems customers have experienced using Windows Server Hyper-V and how the consultants and support persons involved in these situations have helped resolve the customer issues. Because all of these examples are drawn from actual customer stories communicated privately to me by colleagues, I’ve changed some of the details of these stories for the sake of customer privacy and also to generalize the problems and solutions so the stories can be viewed more as lessons learned or best practices advice than support cases. If any readers of this article have similar Hyper-V troubleshooting stories of their own that they’d like to share with our readers, you can either post them here as a comment to this article or email them directly to me at [email protected].
VM Connect won’t connect to Gen2 VM
Bob is running Windows Server 2012 R2 Hyper-V on his host system that is a node in a two-node host cluster. He has created a new Generation 2 virtual machine with a fixed-size disk on the host and has not yet installed a guest operating system on the virtual machine. He starts the new virtual machine and opens the VM Connect tool on the Hyper-V host to connect with the virtual machine. The VM Connect window displays the message “Trying to connect to VM” and then it freezes, preventing him from turning off or shutting down the virtual machine or even closing the VM Connect tool. In fact to close the VM Connect tool Bob has to open Task Manager and kill it.
Bob tries to investigate what’s going on by seeing if any other issues might be occurring with the Hyper-V host. He tries creating a second virtual machine, but the Hyper-V Manager console is unresponsive. He opens the Services.msc snap-in in the MMC console and tries to stop the Hyper-V Virtual Machine Management Service (VMMS) and it displays “Stopping” but never finishes. He tries doing the same using the SC.exe command and once again the service doesn’t stop. He then decides to reboot the Hyper-V host itself and discovers that it won’t shut down!
So he ends up having to physically turn off the host system. He then starts the system in Safe Mode and everything seems OK. So he starts it again in Normal Mode and is able to open the Hyper-V Manager console. Thinking that maybe he should have mounted an ISO for a guest operating system on the VM before trying to start it, he does this and starts the VM again and tries once again to connect with it using the VM Connect tool, but the same problem occurs.
Bob performs a hard restart once again on the host and checks for any new updates that should have been applied from Windows Update. The Windows Server installation is reported as being fully up to date with patches. He tries creating a Generation 1 virtual machine on the host and, lo and behold, he is able to connect to it using the VM Connect tool. So it seems that only Generation 2 virtual machines exhibit this strange behavior.
Bob finally consults an expert in the consulting field who works with Hyper-V customers and asks him for any advice he might have concerning his problematic situation. After further investigation by the expert and Bob working together, several things are tried but no resolution is found for the problem. Finally Bob starts examining the state of other hardware on the host system including the system motherboard, storage hardware, and onboard network card. He discovers that newer firmware is available for the motherboard and an updated driver is available for the 10GbE network adapter. After updating both these hardware items the problem involving Generation 2 virtual machines was resolved.
Hyper-V troubleshooting: Lessons learned
It’s easy to forget when you are maintaining a Windows Server system that it’s not just the operating system that needs to be kept up to date. Each of the individual hardware components of the system that has either firmware or a device driver associated with it should also be checked periodically to ensure its associated software is up to date. With client PCs it’s easy to get by with not bothering to flash new BIOS updates or upgrade to newer device drivers when everything is still working properly and the user of the computer hasn’t complained about experiencing any problems with it. But with high-end server systems such as rack or blade servers used for running Hyper-V host clusters, it’s not uncommon for the manufacturer to release several firmware and driver updates in the months following the release of the new system onto the market.
Maybe this is just another symptom of our push in today’s society to get products out the door as fast as possible and then move on to working on the next version of the product, or maybe it’s just the limited nature of human attention span. Either way, as part of your Hyper-V troubleshooting — or any troubleshooting —make sure you keep track of any new firmware or driver updates that come out for your high-end server systems because, as this article illustrates, some strange things can happen if a hardware component is not working exactly as designed on the system.
As a final aside, there’s another issue with Generation 2 virtual machines on Windows Server 2012 Hyper-V hosts that can manifest itself when you try and use pass-through disks as the storage for your virtual machines. This issue is documented in Microsoft Knowledge Base article KB3102354 and fortunately there’s a hotfix available for it as the article describes.
Photo credit: Wikimedia