World gone wrong: Using event logs to troubleshoot Hyper-V

The event logs in Windows Server are one of the first places you should look when something goes wrong. Navigating them can be challenging, however, for several reasons. First, there are a lot of different event channels in Windows Server and the ones displayed will depend upon not only on which server roles and features you have installed but also on which version of Windows Server you’re working with. Second, the names given to event channels by Microsoft are often cryptic and unusual. Third, the Event Viewer administrative console is somewhat clunky to work with so you’re better off using Windows PowerShell to grab and analyze events. That means, of course, that you’ll need some proficiency with PowerShell to work effectively with the event logs. Fourth and finally, Microsoft’s online documentation of event log channels for Windows Server is somewhat lacking, to say the least. This article examines some of the event channels you might want to use when you need to troubleshoot Hyper-V hosts and the virtual machines running on these hosts. In a future article, we’ll examine how you can use PowerShell to get events from event logs and glean useful information from them.

Hyper-V event channels

When an event of some sort occurs on a Windows Server system such as a Hyper-V host — for example when a virtual machine shuts down on the host — the event is written to the appropriate event log channel. Basically, you can picture an event log channel as a kind of sink that traps certain kinds of events happening on the system. The trapped events can then be read using an event log collector tool such as the built-in Event Viewer tool. The trapped events can also be collected and analyzed by using Windows PowerShell cmdlets like Get-EventLog and Get-WinEvent.

Hyper-V, of course, is the hypervisor technology that provides virtualization capabilities in the Windows Server platform. Hyper-V was first introduced in Windows Server 2008, but it’s been enhanced over the years with additional features as the Windows Server platform itself has continued to evolve. And as new capabilities have been added to Hyper-V in Windows Server 2012, Windows Server 2012 R2, and Windows Server 2016 there have been corresponding changes in which event log channels are defined on these platforms. The following table lists the available channels for Hyper-V in each version of Windows Server as best I’ve been able to determine based on the limited amount of documentation Microsoft provides:

Note that some of these event channels may not be present on your host if certain features aren’t enabled. For example, the Hyper-V High-Availability event channel requires that the Failover Clustering feature is installed on the host. Note also that certain kinds of event logs are only present in a channel if they have been turned on. Analytic and Debug are two types of logs of this nature. For a brief but helpful explanation of what kinds of events some of these different event channels log you can see this post by Lars Iwer on the Microsoft Virtualization Blog.

Note that event logs also come in various channel types such as Admin, Analytic, Debug, Diagnostic, Operational, Networking, Reservation, and Storage. For example, the figure below shows four types of event logs in the Hyper-V-VMMS event channel. Note also that some of these types of channel are only collected after the administrator has turned on logging for that particular type of channel.

Using Hyper-V event logs for troubleshooting

Apart from the System event log, the Hyper-V-VMMS event log channel is probably where you want to start most times when you’re troubleshooting problems. That’s because the Virtual Machine Management Service (VMMS) on a Hyper-V host manages many of the key functions of such hosts and the virtual machines running on those hosts. For example, when a live migration is initiated an event (information has been abbreviated) similar to the following will be logged in the Hyper-V-VMMS-Admin log:

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Event ID: 20413
Level: Information
User: SYSTEM
Computer: BOX1.CONTOSO.COM
Description: The Virtual Machine Management service initiated the live migration of virtual machine ‘SRV3’ to destination host ‘BOX2’

If the live migration fails, then an error event like the following one is likely to be found in the System event log on the host:

Planned virtual machine creation failed for virtual machine ‘SRV13’: An existing connection was forcibly closed by the remote host. (0x80072746). (Virtual Machine ID). Failed to receive data for a Virtual Machine migration: An existing connection was forcibly closed by the remote host. (0x80072746).

The Hyper-V-VMMS-Admin log will then likely show some additional events like these:

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Event ID: 22038
Level: Error
User: SYSTEM
Computer: BOX1.CONTOSO.COM
Description: Failed to send data for a Virtual Machine migration: An existing connection was forcibly closed by the remote host. (0x80072746).

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Event ID: 21018
Level: Error
User: SYSTEM
Computer: BOX1.CONTOSO.COM
Description: Planned virtual machine creation failed for virtual machine ‘SRV3’: An existing connection was forcibly closed by the remote host. (0x80072746). (Virtual Machine ID).

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Event ID: 22040
Level: Error
User: SYSTEM
Computer: BOX1.CONTOSO.COM
Description: Failed to receive data for a Virtual Machine migration: An existing connection was forcibly closed by the remote host. (0x80072746).

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Event ID: 21024
Level: Error
User: SYSTEM
Computer: BOX1.CONTOSO.COM
Description: Virtual machine migration operation for ‘SRV3’ failed at migration source ‘BOX1’. (Virtual machine ID)

These events recorded in the Hyper-V-VMMS-Admin event log suggest that some sort of network problem has caused the live migration to fail on the Hyper-V hosts. At this point we need to start troubleshooting network connectivity between the two Hyper-V hosts involved in the live migration to try and see what might be wrong. To do this one could start with the usual tools like ping, but it could also be a hardware problem involving a NIC in one of the hosts or it might be a misconfiguration of some advanced networking setting on one of the NICs. Of course there are probably a million thigns that can go wrong when it comes to ensuring network connectivity, but that’s another story.

Here’s another example of using the Hyper-V-VMMS-Admin event log to troubleshoot a problem with Hyper-V hosts. An administrator was seeing the following error event (information has been abbreviated) being logged repeatedly every 10 minutes on his clustered Hyper-V hosts:

Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Event ID: 29120
Level: Error
User: SYSTEM
Computer: HOST1. CONTOSO.COM

The event also included the following info:

0x80070002
C:\ClusterStorage\volume4\hyper-v\

The administrator was puzzled by the reference to “volume 4” since that particular CSV volume didn’t exist on his Hyper-V host cluster. A quick search online for the error code might “0x80070002” suggest that the problem has something to do with Windows Update, but this turns out to be somewhat misleading. Instead, it turned out that the problem was associated with the Hyper-V Replica Broker. It turned out that the administrator had renamed all of the CSV volumes on the cluster to adhere to their standing nomenclature, but the configuration for the Broker still pointed to an old volume. Once he updated the path in the Broker to point to the new name for the CSV, errors for Event ID 29120 ceased to be logged in the Hyper-V-VMMS-Admin event log. This illustrates the importance of understanding how recent changes you might have made to the configuration of your Hyper-V hosts might lead to unexpected errors happening if you modified a setting incorrectly or forgot to make a required change to some setting.

Having a good grasp of the Hyper-V architecture and how its various components interact and work together can also be helpful when troubleshooting problems with Hyper-V hosts. A good example of how such knowledge can be used when a problem occurs can be found in this post by Aidan Finn on how to understand Hyper-V error messages. Aidan is a Microsoft Most Valuable Professional who blogs regularly about Microsoft infrastructure solutions and if you have Hyper-V deployed in your environment I highly recommend Hyper-V administrators follow Aidan’s blogging activities.
Featured image: Shutterstock