Troubleshooting Hyper-V Network Virtualization (Part 1)

If you would like to read the other parts in this article series please go to

Troubleshooting any technology component is not an easy task. You must make sure that you have a very good understanding of the technology before you start troubleshooting. Hyper-V Network Virtualization, sometimes referred to as HNV, is a technology mainly designed for cloud hosting providers. Cloud Hosting providers maintain the virtualized workloads for the customers and need to make sure virtual machines are functioning properly. In case of any issues, cloud hosting providers need to troubleshoot the issues and have services up and running with minimal downtime.

In a hosted DR scenario where customer’s virtual machines are kept at both the places; off-premises and on-premises, it becomes necessary that customers troubleshoot their end of the components themselves. This article explains a scenario where customer virtual machines are located in a Shared IaaS cloud and it is the cloud hosting provider who needs to troubleshoot Hyper-V network virtualization components.

There are two troubleshooting approaches a virtual administrator can follow for troubleshooting Hyper-v network virtualization components; 1) verifying Hyper-V Network Virtualization configuration on all Hyper-V Hosts and 2) enabling tracing on network virtualization components. First troubleshooting approach explains a few basic steps you can follow to figure out issues with the HNV configuration. The second troubleshooting approach is more advanced and can be used if there is nothing wrong with the HNV configuration. We will learn how to check Hyper-V network virtualization configuration on Hyper-V Hosts using PowerShell cmdlets, and how enabling tracing on network virtualization components help you get to the root cause which will be explained in the next part of this article series.

Before we start troubleshooting issues, let’s use a Hyper-V Network Virtualization design which will be used throughout this article series for troubleshooting purposes. As shown in the below figure 1.0, there are four virtual machines running on two Hyper-V hosts (HVHOSTA and HVHOSTB). Let’s assume, blue virtual machines belong to Customer-A and green virtual machines belong to the Customer-B.

Figure 1.0: Hyper-V Network Virtualization Design hosting two customer virtual machines

VSID 5001 is assigned to VM1 and VM3 and these virtual machines belong to the virtual subnet 10.0.0.0/24. VM2 and VM4 are assigned with a VSID 5002. These virtual machines belong to another virtual subnet 10.0.1.0/24. Provider Address (PA) 192.168.10.67 manages virtual machines running on HVHOSTA and PA 192.168.10.68 manages virtual machines running on HVHOSTB.

Tip:
PA is responsible for handling communication between virtual machines located on local or remote Hyper-V Servers.

As shown in figure 1.0 above, VM1 and VM3 should be able to communicate with each other but, somehow the communication is broken between these two virtual machines and you need to diagnose issues and resolve it. For purposes of this article series, we will troubleshoot communication issues between VM1 and VM3.

Verifying Hyper-V Network Virtualization Configuration on all Hyper-V Hosts

Your first task is to verify the HNV configuration on both the Hyper-V Hosts. Hyper-V Network Virtualization configuration can be verified using either SCVMM (System Center Virtual Machine Manager) or HNV PowerShell cmdlets. Since SCVMM uses PowerShell cmdlets to create configuration entries for Hyper-V network virtualization, it is easy to check HNV configuration using PowerShell cmdlets. To make sure Hyper-V Network Virtualization is configured correctly, you can use below PowerShell cmdlets which are explained shortly.

Get-NetVirtualizationCustomerRoute
Get-NetVirtualizationProviderAddress
Get-VMNetworkAdapter
Get-NetVirtualizationLookupRecord

Before you start to verify HNV configuration on all Hyper-V Hosts, it is a good practice to check which deployment method was chosen to implement Hyper-V Network Virtualization policy settings. There are two ways to implement Hyper-V Network Virtualization; using SCVMM or Network Virtualization PowerShell cmdlets. A small cloud hosting provider will use PowerShell cmdlets to implement network virtualization policies for the customer virtual machines. You need to keep in mind that, in case PowerShell cmdlets are used to implement HNV, the configuration is lost when Hyper-V server reboots.

Running Get-NetVirtualizationLookupRecord PowerShell cmdlet on a Hyper-V Host will tell you if you have any policy entry configured or not. If output is blank or nothing is returned, then you must rebuild the Hyper-V network virtualization using PowerShell cmdlets if you did not use the SCVMM deployment method. Many virtual administrators design PowerShell scripts and run them to rebuild Hyper-V network virtualization configuration on all Hyper-V Hosts when needed. As part of best practice, you must ensure that PowerShell scripts are created and executed after the Hyper-V servers restart. This ensures the network virtualization configuration is rebuilt after a server restarts.

In case you have used SCVMM deployment method to implement Hyper-V network virtualization, there could be multiple reasons as to why VM1 cannot communicate with VM3 on HVHOSTB as explained below:

Virtual Machines are not assigned or assigned with a wrong Virtual Subnet ID (VSID)

A virtual machine participating in Hyper-V Network Virtualization must be assigned with a Virtual Subnet ID (VSID). A virtual machine does not participate in the Hyper-V Network Virtualization if a VSID is not assigned. To list VSIDs assigned to VM1 and VM3, run the below PowerShell commands on HVHOSTA and HVHOSTB hosts:

Get-VMNetworkAdapter | FT VMName, VirtualSubnetID -AutoSize

As shown in the above output, VM1 is assigned with the correct VSID (5001) which is absolutely fine. In case output returns a wrong VSID for VM1, use Set-VMNetworkAdapter command to assign the correct VSID to VM1.

Execute the same command on HVHOSTB to make sure VM3 is assigned with the same VSID (5001). The output should return similar to below on HVHOSTB:

Lookup Records are incorrect for VM1 and VM3

Lookup Record is what WNV module uses to reach out to virtual machines located on the local or remote Hyper-V Servers. Lookup records are created using Set-NetVirtualizationLookupRecord PowerShell cmdlet. Each lookup record is created with a Provider Address which tells each Hyper-V Host where to send the traffic for each virtual machine participating in the Hyper-V network virtualization. A lookup record looks like below:

VMName VM1 -CustomerAddress “10.0.0.99” -ProviderAddress “192.168.10.67” -VirtualSubnetID “5001” -MACAddress “00155D013202” -Rule “TranslationMethodEncap”

Tip:
The lookup records are created for every virtual machine participating in the Hyper-V network virtualization and must be populated on all Hyper-V hosts hosting the customer virtual machines.

To check the Lookup records on HVHOSTA and HVHOSTB, execute the below command on both the Hyper-V Hosts:

Get-NetVirtualizationLookupRecord | FT VMName,MACAddress,VirtualSubnetID,CustomerAddress,ProviderAddress –AutoSize

The command should return the below output on both the Hyper-V hosts:

Since we are troubleshooting communication issues between VM1 and VM3, make sure to check the following items for VM1 and VM3 in the above output on both the Hyper-V hosts.

MAC Addresses for both the virtual machines are correct. MAC Addresses can be assigned either statically or dynamically. Give a very close look at this part to ensure MAC Addresses are correct for both the virtual machines. Since SCVMM ensures that each virtual machine is assigned with a unique MAC Address, there is no need to pay attention here if HNV configuration is deployed via SCVMM.
VirtualSubnetIDs assigned to VM1 and VM3 are correct. VM1 and VM3 must be assigned with VSID 5001 which is correct as per the output shown above.Tip:
It is imperative to understand that Lookup Records are also created with the correct VSID of the virtual machine which is completely different from assigning a VSID to a virtual machine.
Provider Addresses assigned to VM1 and VM3 are correct. For VM1, PA should be 192.168.10.67 and for VM3, PA should be 192.168.10.68. As per the output above, VM1 and VM3 are assigned to correct provider addresses.

In case you see any issues with the above output, re-populate lookup records on both the Hyper-V hosts using Set-NetVirtualizationLookupRecord PowerShell cmdlet.

Summary

The first part of this article series highlighted two approaches you can follow to troubleshoot issues with Hyper-V Network Virtualization. The first approach is to verify the Hyper-V network virtualization configuration on all Hyper-V Hosts using PowerShell cmdlets, and second approach can be used to figure out issues by enabling tracing on the virtualization components. In this article, we also learned that Hyper-V network virtualization configuration is lost when a Hyper-V server restarts if HNV configuration is implemented manually using PowerShell cmdlets.

In the next part of this article series, we will continue to focus on other possibilities which might cause communication failures between VM1 and VM3 and also touch upon the second approach which is to enable tracing on virtualization components to get to the root cause.

If you would like to read the other parts in this article series please go to