Back in the days when Exchange was run only on physical servers, some companies dedicated hardware exclusively for Exchange servers. Nothing wrong with that. Virtualization, made popular by Hyper-V, VMware, and others, has been around for quite a while now, making it possible to run Exchange VMs on these virtualization platforms. While it is easy to over-spec a server on one of these virtualization platforms, they do come with there own flaws. (Please don’t take this to mean I am against virtualization.) Flaws can be one of the following:
- Host lockups
- Network access lost
- Purple screens on VMware
- Failover not working on hosts
- Storage lockups
- Driver issues on either platform
- Cluster problems
What I have noticed is that the newer operating systems are more resilient than the older ones. Let me expand a bit. Server 2008 R2 that runs Exchange 2010 has seen its fair share of issues. This is not platform-specific or vendor-specific. Mailbox servers, as an example, take a long time to shut down on Server 2008 R2. Booting also takes a long time.
Another example is drivers. Hyper-V doesn’t have the option to allow RSS (receive side scaling) but in VMware, with the VMXNET3 driver, you can enable it for better performance on the network.
On VMware 6.7 as an example, I have seen servers lockup on 2008 R2 servers that are on a compatibility version of 14. You cannot launch anything within the operating system. Anything elevated just does not open and the server sits in a state and says it needs to be activated. If you shut down the server and migrate its compatibility to V15 and startup, all the symptoms disappear. Take note: This does not happen to every 2008 R2 server, it happens randomly.
Other issues seen on both Hyper-V and VMware is the blue circle on the network cards. Here is a list of items I have tried that works on some servers and not others:
- Rebooting into safe mode and uninstalling VMware tools on that platform and on Hyper-V removing the NIC and letting Windows pick it up again. (uninstalling or installing applications in safe mode require a command to be run to allow the Windows installer service to be accessible).
- Resetting the TCP IP stack using the netsh int ip reset command and rebooting.
- Removing the config file for the network in the registry and then opening the network properties and clicking OK re-creates it and after a reboot, the blue circle is gone.
- Running an SFC /Scannow command to verify the integrity of the operating system.
- Buffer overrun in VMware. Increasing the Ring 1 and 2 options on the NIC to 4096 and the small RX to 8192. (Hyper-V does not have this option.)
- Using a VMXNET3 driver on Server 2008 R2 does not always work. The OS simply does not notice the NIC even with the driver and an E1000 is the only option that works.
- Window Server 2008 R2 only logs in with temporary profiles and your desktop is limited to what you can do.
VMotion networks with an IP conflict can also cause issues in the virtual machine. A symptom I have seen is that the stores go into a disconnected state and the return but the copy queue length just keeps growing and not flushing the logs or committing them to the database. Another problem I have noticed on both platforms is the yellow exclamation mark on the NIC. This is problematic especially when your domain controllers cannot send out DNS requests and so you end up with a queue buildup on hubs.
The only real way to resolve this is to reboot the server. Disabling and enabling the NIC simply does not work. Traffic still does not go out.
The next issue I noticed is that it looks like a machine is running (and it is) but you cannot ping the server from another server and you cannot ping out from the server. As Exchange is Active Directory dependent, the server lists loads of event logs that domain controllers cannot be contacted.
The above is a broad list of items. In upcoming articles, I will talk about RSS and RX settings in more detail and how to configure them and the benefit of doing so. You may be wondering, why are you running Exchange 2010 on Server 2008 R2 when you can install it on Server 2012? Well, the reason is if you use a vendor application that runs on top of Exchange, almost like the authoritative source, they only support certain versions of the operating system and so you can only run with that version until you upgrade to Exchange 2016 on Window Server 2016, which has RSS and the other features available to the operating system.
Although Windows Server 2016 and 2019 might be more resilient than Windows Server 2008 R2, it can also give you headaches. If your storage is acting up, this operating system seems to show it more aggressively than legacy operating systems. The operating system will be working and the next minute anything you click in the OS just gives you a white screen and upon rebooting you are told that the operating system is not valid or missing.
Exchange VMs on virtualization platforms: Avoid the headaches
Windows Server 2016 also seems to boot into a repair mode if the host has an issue or something triggers a blue screen and forces the server to reboot. Each platform, as you can see, can work for you but can also give you headaches. Patching the underlying infrastructure is critical, no matter what you are running, not only from a security perspective but from bugs that are fixed to enhancements for virtual machines.
Featured image: Shutterstock