I like hanging out with colleagues from the IT profession because they’re usually great storytellers. That’s probably because those of us who live down in the trenches most of the time tend to experience IT as it really is: raw and unretouched. “Trench tales” is what I call their stories, and while they often start out as horror stories, they usually have a happy ending. Below is one such story that tells about an Exchange Server that unaccountably seems to have been missing a good percentage of its available disk space. I’ve fictionalized some of the details of this story to hide the individual and organization involved. But this shouldn’t hinder interested readers from gleaning much helpful troubleshooting tips from hearing about how this systems administrator managed to figure out what was wrong with his setup. And as this present story indicates, the answer is often staring you in the face.
Anyway, I hope this story and future trench tales I’ll be sharing here on TechGenix will benefit many of our readers. And if you have a story of your own you’d like to share to help out other IT pros, feel free to email me at firstname.lastname@example.org.
The math just didn’t seem to add up properly. Bob had a Windows Server 2012 R2 instance installed on a dynamically expanding virtual hard disk with the default maximum size of 127GB in a virtual machine running on a Hyper-V host. When he opened Windows Explorer and examined C: drive in the guest operating system, it showed only 15GB remaining free space for the volume. But when he used Explorer to select all of the files and folders in the volume and then right-clicked and selected Properties, it reported that only 45GB of disk space was being used. And yes, he had deselected the “Hide protected operating system files” checkbox on the View tab of the Folder Options dialog in Explorer.
Still puzzled, Bob decided to try using a third-party disk space reporting tool called WinDirStat to find out what was happening here. I used this tool myself with good results a couple of years ago when I discovered that the hard disk on a Windows 7 machine I was using seemed to be quickly filling up for no reason I could think of. It turned out that the Office Document Cache on my machine had been filling up with tens of gigabytes of FSD files hidden away in the user profile folders on my machine, and using the Microsoft Office Upload Center allowed me to delete these cached files and recover some much needed disk space. I shared this story in some detail in Issue #1026 of my popular WServerNews weekly newsletter.
Anyway, Bob decided to follow a similar approach of troubleshooting disk space issues using the WinDirStat utility. What he discovered was that WinDirStat reported an item which it labeled “Unknown” as consuming over 50GB of space on the virtual disk. He tried opening the Properties of this item, but nothing useful was displayed. He wasn’t able to delete the item either as the Delete button in the utility’s toolbar was grayed-out when the unknown item was selected.
Where’s my disk space: Some solutions tried
What could be eating up such a massive amount of disk space on his virtual machine? Bob needed to recover some of that space so his server could utilize it, so he started by trying a couple of things. First he tried defragging the disk using the following command:
The /L switch was introduced in Windows Server 2012 and performs a retrim operation on the specified volumes, which can help recover unused disk space on thinly provisioned volumes including dynamically expanding virtual disks. This had no effect on the disk space results reported, so next Bob tried running the following command:
Vssadmin list shadowstorage
This command can be used in Windows Server to list all shadow copy storage associations on the system, and he thought if anything turned up this way he could use the Vssadmin delete shadowstorage command to delete those associations. But this didn’t work either, so next Bob tried this command:
Chkdsk /sdcleanup /f
This command runs Chkdsk and garbage-collects any unneeded security descriptors (ACLs) that are found on the system. Bob didn’t really expect anything significant to happen when he ran this, and sure enough he wasn’t disappointed.
Next Bob tried running the File System Utility command Fsutil like this:
fsutil volume allocationreport C: > report.txt
The idea behind running this command, whose allocationreport switch was first introduced in Windows Server 2012, is to collect detailed information about how much space is used by hidden metadata files on the specified disk volume and send it to a text file as a report so you can examine it. Unfortunately, the report that running this command had generated indicated only that several tens of gigabytes had been allocated to the default stream under User Files.
Bob had now reached an impasse, so he reached out for some help.
Bob told me later that a colleague of his had made a simple recommendation in response to his plea for help: try running these disk space sniffing utilities as Administrator.
It turns out that commands like fsutil and utilities like WinDirStat often won’t report disk space usage correctly unless you run these tools using a user account with administrator privileges on the system you’re examining. You may even need to take ownership of certain folders like the System Volume Information folder to properly determine how disk space is being used by such folders, and of course that’s not a good idea as modifying system ACLs can prevent Windows from working properly.
The moral of the story is that administrative privileges are often required for system utilities to function properly when you use them on Windows. Unfortunately, I can’t tell you what Bob actually found was the culprit that was eating up his disk space, but running WinDirStat quickly led him to track this culprit down.
There are several articles I’ve found that can be especially helpful when you’re trying to track down missing disk space problems. Here are some of the most useful ones:
Where Did My Space Go? - An article by Jeff Hughes on the Ask The Core Team blog on TechNet.
What’s the deal with the System Volume Information folder? - A brief but informative overview by Raymond Chen on his blog The Old New Thing.
NTFS Misreports Free Space (Part 3) - An article on the Ntdebugging Blog about the Fsutil volume allocationreport command.
What’s filling up your disk? - The aforementioned issue of WServerNews that describes how I used WinDirStat to solve my own disk space problem.
IT salaries: good, bad or ugly? - Check the Mailbag section of this issue of WServerNews for some reader suggestions about other disk space usage reporting tools you may find useful.
And feel free to recommend your own disk space usage tools using the Comments feature of this article.
Photo credit: Wikimedia