If you would like to read the other parts in this article series please go to:
Introduction
In the second part of this article series, I discussed a situation in which Windows had misidentified a network adapter. The driver that Windows had attempted to use was similar enough to the correct driver that network connectivity worked at times, but was intermittent. Although the driver truly was to blame in that situation, there is another issue that I have encountered from time to time that produces symptoms that are nearly identical.
About a year ago I had a project which required the creation of about ten or eleven virtual machines. Rather than spending big bucks on a single server that could comfortably host all of the necessary virtual machines, I decided to save some money by purchasing three lower end servers instead.
When the new servers arrived I installed Windows Server 2008 R2, joined the servers to the domain. Up to this point in the process everything seemed to be working just as it should. However, strange things started to happen after I installed the Hyper-V role and began creating virtual machines.
I first noticed the problem when I tried to connect to a virtual machine using Remote Desktop. Windows took forever to launch my Remote Desktop session. Less than two minutes after I signed into my virtual machine, it dropped my connection.
Although I attempted to reconnect to the virtual machine, I was simply unable to. I also tried pinging the virtual machine, but all attempts were unsuccessful.
Since I wasn’t able to ping the virtual machine, I tried pinging the host server. My first attempt was successful but all subsequent attempts failed.
At that point I decided to walk to the next room to see what was going on with the server. The NICs all showed a status of connected, but connectivity was intermittent at best.
I knew that the server’s networking components were configured correctly. After all, I had been able to successfully join a domain. Because I knew that the software was configured correctly and I also knew that the server was brand new, I assumed that I had a bad network card, a bad patch cable, or both.
On that particular day I had a lot of work to do, so I really didn’t want to waste any more time trying to diagnose the problem. In the interest of saving time I replaced both the network card and the patch cable. I also connected the new patch cable to a different port on my network switch just in case a switch port had gone bad.
As luck would have it, my repairs made absolutely no difference. The server was still experiencing all manner of network connectivity issues. This made me wonder if perhaps a component within the TCP/IP stack had become corrupted.
Rather than attempting to further troubleshoot the problem, I made the decision to reformat all of the server’s volumes and start from scratch. I reinstalled Windows and then applied all of the latest patches from Microsoft. Before I wasted any time trying to join the server to my domain I decided to perform a series of diagnostic tests to make sure that my network connectivity was working.
All of the tests were successful. Confident that I had solved my problem I joined the server to my domain and performed a couple more quick tests. Everything still appeared to be working normally, so I went ahead and installed the Hyper-V role. After doing so, the problems came back almost immediately.
At that point I was relatively sure that Hyper-V was causing the problems. Even though I had never run into this type of issue with Hyper-V before, it made sense that Hyper-V could be to blame. After all, Hyper-V places a layer of abstraction between the host operating system and the network adapter.
Normally Windows Server 2008 binds several networking components directly to the network adapter. These networking components include the Client for Microsoft Networks, the QoS Packet Scheduler, File and Print Sharing for Microsoft Networks, and Internet Protocol Version 4 (TCP/IPv4). Depending on the way that the network is configured there might be additional components bound to the network adapter, but these components are the ones used by default.
When Hyper-V is installed and configured to use a network adapter, Windows breaks the bindings between the network adapter and the components that I mentioned a moment ago. The Hyper-V role creates a virtual network switch that is bound to the network adapter. In fact, if you look at the adapter’s properties through the host operating system you can see that the only component that is bound to the adapter is the Microsoft Virtual Network Switch protocol. You can see what this looks like in Figure A.
Figure A: The only component that is bound directly to the network adapter is the Microsoft Virtual Network Switch protocol.
Of course the host operating system still needs a way of communicating across the network. That being the case, Hyper-V creates a virtual network adapter within the host operating system. All of the components that had previously been bound to the physical network adapter are bound to the virtual network adapter. If the server was configured to use a static IP address then the address is removed from the physical network adapter and assigned to the virtual network adapter.
The host operating system communicates through the virtual network adapter as if it were a physical adapter. The virtual network adapter passes the outbound packets through the virtual network switch, which routes the packets through the physical network adapter and then out to the physical network.
I am telling you all of this as a way of illustrating the point that the Hyper-V role makes major changes to the server’s network architecture even before you create the first virtual machine. Since I knew that my server was having no trouble with network connectivity prior to installing the Hyper-V role, I knew that some aspect of the virtual network infrastructure had to be causing my problems.
After spending many hours troubleshooting the problem, I eventually determined that the problem was related to my network adapter’s Large Send Offload (LSO) feature. When Windows needs to send a large chunk of data it will typically create a series of packets, each of which contains a portion of the data that is being sent.
Because the packet creation process consumes CPU resources, some higher end network adapters try to offload the packet creation process to the hardware. Rather than requiring Windows to create numerous packets for large data streams, Windows creates much larger packets that can be up to 64 KB in size. When the packets reach the network card, it breaks the packet into a series of frames that can be sent across the network.
Normally the Large Send Offload feature works really well. However, the layer of abstraction that Hyper-V places on top of the network adapter causes the Large Send Offload feature to break. The solution to the problem is to simply disable this feature.
To disable the Large Send Offload feature, open the Control Panel within your host operating system and go to Network and Sharing Center | Change Adapter Settings. Next, right click on the listing for your physical network adapter and select the Properties command from the resulting shortcut menu. When Windows displays the network connection’s properties sheet, click the Configure button. Windows will now display the properties sheet for the network adapter. Go to the properties sheet’s Advanced tab and locate the Large Send Offload (IPv4) option. Finally, set the value to Disabled, as shown in Figure B. Click OK to complete the process.
Figure B: You may have to disable the Large Send Offload feature.
Conclusion
As is the case with any other application there are any number of things that can go wrong with Hyper-V. In most cases however, the problem is relatively easy to correct.
If you would like to read the other parts in this article series please go to:
