Taking Control of VM Sprawl (Part 4)

If you would like to read the other parts in this article series please go to:

Introduction

At the end of my previous article, I said that I would use this article to share with you some strategies for identifying virtual machines that have been abandoned. Let me just say up front that identifying abandoned virtual machines can be REALLY tough to do. That’s why I put so much emphasis on documentation in the previous articles. If you start out by documenting the known virtual machines, you can narrow down the number of virtual machines that might possibly be abandoned.

This process of elimination works really well as a starting point. Even so, it will only get you so far. Furthermore, there is no one, single technique that you can use to identify abandoned virtual machines with 100% accuracy. No matter how you go about the task, it is going to be a tedious manual process.

It is also worth noting that your options for identifying abandoned virtual machines are going to vary widely depending on the hypervisor that you are using and on any management software that you might have in place. Different hypervisors offer different capabilities. The same thing can be said for management software. That being the case, the first thing that you should probably do is to identify features within your hypervisor and any management software that you might be using that can help you to potentially identify abandoned virtual machines.

So with that said, I want to begin talking about some general-purpose techniques that you may be able to use in some situations.

Using the Event Logs

Hypervisors typically perform some sort of diagnostic logging. The logs themselves are proprietary to each hypervisor however, and there isn’t really any universal standard for hypervisor logs. Even so, the logs can be a good place to start gathering information about the virtual machine. For the purposes of this article, I am going to be using Microsoft’s Hyper-V and the Windows event logs.

You can open the Windows Event Viewer by opening Server Manager and then selecting the Event Viewer command from the Tools menu. When the Event Viewer opens, you can access the Hyper-V logs by navigating through the console tree to Event Viewer (Local) | Application and Services Logs | Microsoft | Windows. You will notice that this path doesn’t make any mention of Hyper-V. The reason for this is that in Windows Server 2012 R2, there are eleven different containers related to Hyper-V. Each of these containers store a collection of logs related to some specific aspect of Hyper-V.

If your goal is to determine whether or not a virtual machine has been abandoned, then my advice is to expand the Hyper-V VMMs container (which is located beneath the Windows container). The Hyper-V VMMS container stores logs related to the Virtual Machine Management Service. In other words, when someone creates a virtual machine, this is where the corresponding log entry will be placed.

You can see an example of a virtual machine creation event in Figure A. As you can see in the figure, Windows doesn’t give us much to go on. The user is listed as System, so we can’t tell who actually created the VM. We can however, tell exactly when the VM was created, and that might give us a clue as to who created the VM or what its purpose might be.

Image
Figure A: This is the log entry that is generated when a new virtual machine is created.

Incidentally, looking for a record of a virtual machine creation can be like looking for a needle in a haystack. In a production environment, it is a good idea to use a filter. To do so, click on the Filter Current Log link. Virtual machine creation is logged with an event ID of 13002. As such, you can enter this ID into the Filter Current Log dialog box, as shown in Figure B. Upon clicking OK, you will be presented with a record of VM creation, as shown in Figure C.

Image
Figure B: Enter 13002 as the Event ID.

Image
Figure C: The filtered view shows only virtual machine creation events.

This actually brings up an important point. If you look back at Figure A, you will notice that the event description lists Computer as Lab4.MGMT.com. This is the server on which the virtual machine was created. As we all know however, virtual machines are completely portable. If no creation record exists for the virtual machine in question then it could be that the virtual machine was created on another host and then live migrated to the current host. Another possibility is that the virtual machine was imported rather than being created in the usual way.

This raises a couple of questions. First, how can we find a creation record for the VM if that VM was live migrated from another server? Second, how can we get a complete history of the VM?

These two questions are actually somewhat intertwined. Determining a virtual machine’s creation date gives us a good starting point for figuring out if the virtual machine has been abandoned or not, but the creation date certainly doesn’t tell us everything. Often times there may be other tell-tale clues scattered throughout the event logs.  For example, if no significant log entries have been created for the VM for quite some time then it might be a hint that the VM is no longer being actively used. Conversely, if you find that someone attached a new virtual hard disk to the virtual machine last week, then it is an indication that the virtual machine is still being used.

In order to get a virtual machine’s full history, you will have to scan the event logs for every host server that the virtual machine has ever resided on. This can be a tall order to say the least.

If you are using Hyper-V then I would recommend looking into third party event log monitoring tools. There are some really great third party tools available in the market that can help you to create a consolidated view of the event logs from all of your servers. Such a tool would then be able to parse all of those logs and provide you with a report of the relevant information from the various event logs.

If you happen to be working in a VMware environment, then it may be useful to use a different type of third-party tool. Dell offers a product called Foglight that includes a utility for finding virtual machines that have been abandoned. The tool works by looking at virtual machines that have been removed from vCenter but that are still running on one of the host servers.

Conclusion

If you are using Hyper-V and a third-party tool simply isn’t in the budget then you are not completely out of options. You can create your own tool. It is possible to use PowerShell to parse the event logs that are stored on multiple servers, and compile the relevant events. By using this technique, you could theoretically assemble a complete picture of the virtual machine’s entire existence. I will begin showing you how to create such a tool in the next article in this series.

If you would like to read the other parts in this article series please go to:

Leave a Comment

Your email address will not be published.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top