Troubleshooting Storage Using Event Logs
The Windows event logs are one of the first places you should look for clues when your systems or applications are not behaving in the way you expect them to behave. In the old days of Windows Server 2003 there were only a few event logs you had to check with the System, Application and Security logs being the main ones. Then beginning with Windows Server 2008 the number of different event logs and types of logs jumped into the hundreds, and although you can filter and search these logs using Event Viewer and Windows PowerShell, many long-time server admins still often feel overwhelmed by the sheer amount of information Windows Server logs and how to discover and identify what might be relevant for the particular scenario they are trying to optimize or troubleshoot.
To illustrate how to use the Windows Event Logs to troubleshoot storage issues in a Hyper-V environment, I'm going to share with you a short except from the unedited draft of my new ebook Optimizing and Troubleshooting Hyper-V Storage (Microsoft Press, 2013). The content for this topic was contributed by Thomas Roettinger, a Program Manager with the Partner and Customer Ecosystem Team at Microsoft. Thomas and his team work with technologies like Hyper-V and System Center Virtual Machine Manager. His team also runs the Windows Server TAP Program and collects very early technology best practices. Before Thomas joined the Product Group he was the EMEA Virtualization Lead in Microsoft Premier Field Engineering. During this time he was responsible for various Services like the Hyper-V Risk Assessment Program and other Services like the Implementing Hyper-V Workshop. He has rich experience in Cloud implementations across various business segments like Hosters and Enterprises. Thomas maintains a personal blog at http://blogs.technet.com/b/cloudytom but also contributes to his team blog at http://blogs.technet.com/b/wincat.
Hyper-V storage event logs
Windows Server provides several different event log categories you need to look for Hyper-V related issues. You can open the event log viewer either through the modern UI by using the hotkey Windows Key & X, or by launching the mmc snap in directly by typing 'eventvwr.msc'.
You need to expand 'Application and Services Logs', 'Microsoft' and 'Windows' you will find specific application and services logs for windows components. For example you will only see the Hyper-V logs when the Hyper-V Role is installed.
The storage related logs are called:
- Hyper-V-SynthFC (Virtual FC Adapter)
- Hyper-V-SynthStor (SCSI Controller)
Figure 1: A screenshot of the Event Viewer console that shows the different event logs when you select Windows under Microsoft under Applications And Services Logs in the console tree.
When you start a virtual machine, a virtual machine worker process (VMWP.exe) is launched per VM. This log 'Hyper-V-Worker' provides all necessary information related to start, stop action and the run time of a virtual machine. The worker process connects to the virtual machine management services (VMMS). In the example below you will notice for troubleshooting you can look at several Hyper-V logs and sometimes it is required to find the root cause of an outage.
Example: missing virtual hard disk
Patricia is an Administrator and she receives an error message when she tries to start one of her virtual machines but did not pay attention to the error message:
Figure 2: A screenshot showing a dialog box with the message "MissingDisk failed to start."
Patricia is opening the event viewer and checks the related Hyper-V event logs. She looks at 'Hyper-V-VMWP' logs first because the issue happened when she tried to power on the virtual machine.
Event ID 32902
"Missing Disk" Synthetic SCSI Controller: Failed to Power on with Error 'The system cannot find the file specified.'
Patricia next looks at 'Hyper-V-SynthStor' logs to see if she can find more information
Event ID 12240
'Missing Disk': Attachment" could not be found due to error 'The system cannot find the file specified.'
Patricia now needs to identify which file is missing, so she checks the 'Hyper-V-VMMS' logs
Event ID 32902
The absolute path 'd:\missingdisk.vhdx' is valid for the "Hard Disk Image pool, but references a file that does not exist.
Patricia knows that a virtual hard disk file is missing on the D: drive. She is using Windows Explorer to find the file. She now discovers that the entire D: drive is missing:
Figure 3: A screenshot of Explorer with the Computer node selected. Only one hard drive (C: drive) is visible.
This error could happen if you lose storage connectivity no matter how it is presented to your host. There are techniques to provide multiple paths to your storage when using SMB, FC or ISCSI using MPIO or SMB multichannel to provide availability.
In this section you have seen the available Hyper-v event logs for storage. But you also heart about their relationship to other important Hyper-V logs like the Virtual Machine Worker Process and the Virtual Machine Management Service. This is just one example but there are many more scenarios like wrong permissions on a virtual hard disk file (Event ID 12290).
Example: unsupported Fibre Channel adapter
Patricia is an Administrator and she needs to configure a virtual machine with a virtual fibre channel adapter. The Hyper-V host has a fibre channel adapter installed and Patricia is opening the "Virtual SAN Manager" to create a new Fibre Channel SAN. She notice that she is not able to select the physical HBA and the status is "The device or driver does not support virtual Fibre Channel":
Figure 4: A screenshot showing the Virtual SAN Manager properties page with the message "The device or driver does not support virtual Fibre Channel" displayed.
She ignores the messages and continues to add a virtual HBA to the virtual machine. When she tries to start the virtual machine she receives the following error message.
Figure 5: A screenshot showing a dialog box with the message "vHBA failed to start."
Patricia is looking at the 'Hyper-V-SynthFC' event log and discovers the following entry:
Event ID: 32161
'vHBA': Operation for virtual port ……failed with an error: No physical port available to satisfy the request
Patricia opens Windows PowerShell to validate the driver. She does a WMI query to see if the driver does provide support for Hyper-V. She runs the following command:
gwmi –Namespace "root\virtualization\v2" –Class msvm_externalfcport |select *hyper*
The resulting output from this command looks like this:
While this indicates that the driver does not support Hyper-V, Patricia knows that the physical adapter is capable of NPIV and should support Hyper-V. This disparity can happen because of some older driver versions that do not populate the necessary information correctly.
Anyways, after downloading and installing the latest driver from the HBA vendor everything works as expected.
More information on how to troubleshoot storage in Hyper-V environments can be found in my ebook Optimizing and Troubleshooting Hyper-V Storage, which includes content contributed from a number of different experts on the Windows Server team at Microsoft.