Troubleshooting Hyper-V Network Virtualization (Part 3)

If you would like to be notified when Nirmal Sharma releases the next part of this article series please sign up to the WindowsNetworking.com Real time article update newsletter.

If you would like to read the other parts in this article series please go to:

Introduction

In first and second parts of this article series, we learned some of the possibilities which might cause communication failures between virtual machines participating in Hyper-V Network Virtualization. If you do not see any configuration issues and end up doing further research, it is the time to enable unified tracing on the Hyper-V hosts.

When processing an outbound packet, the network traffic leaves the virtual machine, hits the WNV module, and then hits the virtual network switch. On the other hand, when processing an inbound packet, the virtual network switch receives the network packet, sends the packet to WNV module, and then hits the virtual machine. A new NetSh parameter has been added in the Windows Server 2012 and later Operating Systems which can be used to enable the unified tracing to capture the inbound and outbound packets. In this article, we will explain how to use “Unified Tracing” and how it helps you get to the root cause.

As a high-level process, please follow the steps below to troubleshoot issues using “Unified Tracing”:

  1. Enable Unified Tracing on providers to capture events to an ETL file.
  2. Reproduce the connectivity failure issue.
  3. Stop the trace.
  4. Convert the ETL file to a text file.
  5. Finally, analyze the event log file.

Step 1 – Enable Unified Tracing on providers to capture events to an ETL file

The “Netsh trace” context includes predefined sets of trace providers. These providers are known as scenarios which you can enable for troubleshooting. There are two providers available for tracing events related to Hyper-V Network Virtualization; Microsoft-Windows-Wnv and Microsoft-Windows-Hyper-V-VmSwitch providers. Since the network traffic hits both virtual network switch and WNV module, it is necessary to enable the tracing on both the providers. Since only one NetSh trace session can be active at a time, it is necessary to enable tracing on both the providers together. To capture network virtual switch and Hyper-V Network Virtualization events, execute below NetSh command on a HVHOST-A host:

  • NetSh Trace Start Provider=Microsoft-Windows-WNV Level=5 Provider=Microsoft-Windows-Hyper-V-VmSwitch Capture=Yes CaptureType=vmswitch Tracefile=C:\Temp\HNVTrace.etl

The above command enables tracing for both the providers together and any events which are captured when virtual machines communicate with each other, are saved in the C:\Temp\HNVTrace.etl file. As specified in the above command, the tracing is enabled on both the providers; “Microsoft-Windows-WNV” provider for capturing events related to Hyper-V Network Virtualization and “Microsoft-Windows-Hyper-V-VmSwitch” provider to capture events occurring on virtual network switch.

Note:
It is important to understand that enabling unified tracing on HVHOST-A will capture events occurring only on that Hyper-V host. Since VM1 and VM3 are located on different Hyper-V hosts, you might need to enable the unified tracing on HVHOST-B if you do not find any issues at HVHOST-A side after analyzing the tracing log file.

Step 2 – Reproducing the Connectivity failure issues

In Step 2, all you need to do is to reproduce the connectivity issues or the issue which is being investigated. Since we are troubleshooting communication issues between VM1 and VM3, we will use ping utility to check connectivity from VM1 to VM3. Keep in mind that we have enabled tracing on HVHOST-A, so it is necessary that we ping VM3 from within the VM1 Operating System. When we ping VM3 from VM1, all the outbound network packets will be captured in the C:\Temp\HNVTrace.etl file.

Step 3 – Stop the trace

Once you are finished running the Ping command, stop the NetSh trace on the HVHOST-A Hyper-V host. It is important to note that there is no need to spend much time at Step 2. If you do so, you might see a large event tracing file that will be hard to read. So it is recommended that you stop the trace as quickly as possible by executing the command below:

  • NetSh Trace Stop

Step 4 – Converting the ETL file to a text file

In Step 4, you need to convert the ETL file to a text file. An ETL file, sometime referred to as Event Tracing Log, stores the event messages. These messages are stored during one or more trace sessions. Although, you can use Network Monitor 3.3 or similar tools to read the ETL file contents, but you might need to download the Network Monitor 3.3 tool before you can parse the events from ETL file. Since NetSh provides an option for converting an ETL file to text file, it is advisable to do so by executing the NetSh command as listed below:

  • NetSh Trace Convert C:\Temp\NetTrace.etl C:\Temp\NetTraceOutput.txt

Step 5 – Finally, analyze the event log file

Once the ETL file is converted to a text file, open the text file using notepad or similar text editors. There are two types of network packets that can be seen in the event log file; inbound and outbound network packets. The outbound packet processing takes place when Microsoft-Windows-Wnv module receives a packet to be delivered to a virtual machine located on the same or remote Hyper-V host. As you can see in the screenshot below, Microsoft-Windows-Wnv module receives a packet which needs to be delivered to a virtual machine located on local or remote Hyper-V host. Microsoft-Windows-Wnv module specifies that it is processing an outbound network packet as highlighted in the screenshot below:

Image
Figure 1

Similarly, if the Microsoft-Windows-Wnv module receives an inbound network packet, you will see “Inbound Address resolution packet” string in the event log file as highlighted in the screenshot below.

Image
Figure 2

As part of this troubleshooting, we need to find out as to what is causing a network packet to be dropped by the Microsoft Microsoft-Windows-WNV module.

As part of the packet processing, the network packets will be either forwarded or dropped. Before you start to look at the event log file, I would suggest that you look for entries in the event log file, which are related to the VM1 and VM3. What I have seen when a packet is dropped by the WNV module, it logs keywords such as “Dropped” or “Drop” in the event log file. Remember that in Step 2, we used ping command to ping VM3 from VM1 to reproduce the connectivity issue. Since it is an outbound packet, we will need to search for the outbound entries in the event log file.

When Microsoft-Windows-Wnv module drops a packet, it also highlights the reason for dropping the packet. A packet could be dropped for several reasons. One of the reasons is that there is no policy entry configured in the lookup table for VM3 or VM1. Another reason could be that VSID is not matching. As you can see the highlighted text in the screenshot below, since Microsoft-Windows-Wnv module did not find a policy entry in the lookup table, it dropped the packet.

Image
Figure 3

Note:
It is important to understand that you do not see the virtual machine name in the event log file. All you see is the IP Address of the virtual machine.

Similarly, if a packet is dropped because of the mismatch with VSID, Microsoft-Windows-Wnv module will indicate that too in the event log file.

If you find it difficult to search through the event log file to locate the “drop reasons” keyboards, you can use “Find” command to filter the “drop reasons” entries and store in another text file. For example, to capture all “drop reasons” in a separate text file, use the command below:

  • Find /I “Drop reason” < NetTraceOutPut.TXT > AllDropReasons.TXT

Once the above command is executed, all the “drop reasons” entries will be captured in the AllDropReasons.TXT file as shown in the screenshot below.

Image
Figure 4

While it might not be feasible to cover all reasons in one article, but it is true that enabling the unified tracing on Hyper-V host would provide the reason for dropping an inbound/outbound network packets by Microsoft-Windows-Wnv module.

Conclusion

You saw how it is easy to troubleshoot issues with Hyper-V Network Virtualization by using Unified Tracing. The ETL file, when converted into a text file, provides enough troubleshooting data. Since the ETL file contains the reason for dropping a network packet, it becomes easy for virtual administrators to troubleshoot issues with the Hyper-V Network Virtualization by enabling the Unified Tracing.

If you would like to be notified when Nirmal Sharma releases the next part of this article series please sign up to the WindowsNetworking.com Real time article update newsletter.

If you would like to read the other parts in this article series please go to:

Leave a Comment

Your email address will not be published.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top