If you would like to read other parts to this article series please read:
- Resource Allocation in Hyper-V (Part 1)
- Resource Allocation in Hyper-V (Part 3)
- Resource Allocation in Hyper-V (Part 4)
In my previous article in this series, I talked about some of the different hardware considerations that you need to make when allocating system resources to virtual machines in a Hyper-V environment. In this article, I want to continue the discussion by talking about Performance Monitoring in a Hyper-V environment. Although the basic concepts of performance monitoring have changed very little since the days of Windows NT, obtaining accurate performance monitoring information is especially challenging a Hyper-V environment. In this article, I will explain why.
Performance Monitoring
The first thing that I want to talk about is performance monitoring. It seems a little odd to me, but for some reason performance monitoring seems as though it has become the hot virtualization topic almost overnight. I think that part of the reason for this is that people are starting to realize that the Performance Monitor cannot be completely trusted in a Hyper-V environment. It isn’t just Performance Monitor that becomes unreliable though. Many of the other available resource monitoring mechanisms can also no longer be trusted. For example, it is very common for the Hyper-V management console to report completely different levels of CPU usage than what is displayed under the Windows Task Manager. In fact, if you look at Figure A, you can see that the Hyper-V manager is reporting 5% CPU utilization, while the Windows Task Manager is reporting that the virtual machine is using 0% of the CPU resources.
Figure A: The Windows Task Manager and the Hyper-V Manager rarely agree as to how much CPU time a virtual machine is using
The reason for this discrepancy is a little bit complicated, but I will try to keep my explanation as simple as I can. The most important thing to remember when trying to understand the reason for this discrepancy is that Hyper-V allows virtual machines to communicate directly with the server’s hardware. Of course this raises the question that if this is true then why does the Windows Task Manager even show a process for the virtual machine?
The reason why Windows Task Manager shows a process for the virtual machine is because the host operating system has to be able to interact with the guest machine to a minimal degree. Remember that Hyper-V allows you to do things like a snapshoting or saving a virtual machine state. The host operating system uses a worker process to connect to virtual machines so that these types of tasks can be performed. The same process is also used for emulating hardware devices.
OK, so we have established that the Windows Task Manager is completely unaware of how much CPU time a virtual machine is actually using. This raises at least a couple of other questions though. One question is why CPU resources are the virtual machines actually using? Perhaps the more obvious question though, is how can we track that usage?
As I mentioned earlier, Hyper-V allows virtual machines to communicate directly with hardware through the use of the Hypervisor. This is where the vast majority of the CPU usage actually occurs for the virtual machine. You can get a rough idea of how much CPU time a virtual machine is really using by looking at either the Performance Monitor or the Windows Task Manager from within the virtual machine.
If you look at Figure B, you can see that I have run the Task Manager inside of my virtual server. At the time that the screen capture was taken, the Windows Task Manager reports the CPU usage for the virtual machine at 13%, while the Hyper-V Manager only reports the CPU usage at 4% at that same moment.
Figure B: There is a big discrepancy between the Windows Task Manager and the Hyper-V Manager
As you can see, there is a big difference between the level of CPU utilization report by the Windows Task Manager from within the virtual machine, and what is being reported by the Hyper-V Manager. So which one is right? Actually, it is a trick question; both reports are completely wrong, but the Windows Task Manager is a whole lot closer to being right than the Hyper-V Manager is.
The reason why you cannot completely trust the information given to you by the Windows Task Manager is that the performance data that is being reported is skewed. For example, a machine that I got the screen capture off of has a single processor with four cores. As such, there is no dedicated processor servicing the virtual machine. Instead, the virtual machine receives a percentage of the physical hardware’s total CPU time. The virtual machine is not aware of this. It thinks that it has full reign of the system CPU. The CPU usage report provided by the Windows Task Manager would only be accurate if the virtual machine had been allocated all of the system’s CPU time or if there was a dedicated processor servicing the virtual machine. As it is, the Windows Task Manager is actually reporting what percentage of the CPU resources that have been allocated to the virtual machine are actually being used.
So how do you know how much CPU time a virtual machine is actually consuming? I do not think that there is really any way to know for sure exactly how much CPU time a virtual machine is consuming. However, Microsoft has integrated a number of Hyper-V specific counters into the host operating system’s copy of Performance Monitor. Since these counters are Hyper-V aware, you can use them to get a pretty good idea of how the virtual machines are performing.
There are way too many Hyper-V related counters to talk about them all, but there are two counters that you might be particularly interested in. One such counter is the Hyper-V Hypervisor Virtual Processor\% Guest Run Time counter. It is designed to show you how much CPU time a guest machine is really using. For example, if you look at Figure C, you can see that my virtual server was actually consuming about 31% of the server’s physical CPU resources when the screen shot was taken, but the Hyper-V Manager was only reporting a utilization of 7%.
Figure C: The Hyper-V Hypervisor Virtual Processor\% Guest Run Time counter is probably the most reliable way of seeing how much CPU time a virtual machine is using
Another counter that is interesting to look at is the Hyper-V Hypervisor Virtual Processor\% Hypervisor Run Time counter. It shows you how hard the server’s Hypervisor is working to manage the virtual machines. This counter won’t normally be anywhere near as high as the last counter that I showed you unless you have a lot of virtual machines running at the same time. Even so, this counter is worth paying attention to because it does reflect a level of processor usage that is otherwise unreported.
So Which One Do You Use?
In this article, I’ve shown you several different ways of obtaining performance benchmarks for virtual machines, so you may be wondering which method you should use. The answer really just depends on your goal. If you’re trying to find out how much CPU time is actually being consumed, then I would recommend monitoring the Hyper-V Hypervisor Virtual Processor\% Guest Run Time counter. However, if your goal is to find out whether or not a virtual machine’s performance is acceptable for a particular application, then you are best off running the Performance Monitor in the usual way, but from inside the virtual machine. The reason for this is that the resources that are displayed when you run the Performance Monitor inside a virtual machine are the same resources that are available to applications that are running on that virtual machine.
Conclusion
In this article, I have talked about performance monitoring as it related to Hyper-V. In the next article in this series, I will discuss some additional resource allocation issues related to CPU affinity and to other hardware devices.
If you would like to read other parts to this article series please read:
