Updating Microsoft Exchange to a newer cumulative update (CU) is something many admins have been doing frequently over the last few weeks because of the major hack on Exchange 2016 and Exchange 2019 uncovered in March. Cumulative updates, for those new to Exchange, contain the previous list of fixes and updates and also new ones. Cumulative updates are basically full versions of Exchange. If you are running Exchange 2016 or Exchange 2019 and you have Symantec antivirus on the server, you need to take note of a feature in Symantec that seems to block the cumulative update from either copying, installing, or downloading. The feature I am referring to is called “Intrusion Prevention,” also known as IPS. While IPS is great for blocking ransomware attempts or other malware, it seems to perceive that Microsoft ISO files have invalid signatures and simply won’t allow you to do anything with the file. This means you cannot even mount it to run the upgrade or installation on an Exchange Server.
Updating Exchange with Symantec antivirus
How do you update an Exchange server with this issue? I spent a few hours troubleshooting and found that by uninstalling that option only and leaving the rest of Symantec intact, you can download, copy, and mount the Exchange 2016 or Exchange 2019 ISO without issues. The network speed copy also drastically improves from a few KB to 400MB/sec, depending on how fast your backend infrastructure is. I have found that when you remove this component, your copy should return, but you need to reboot. Even though it does not prompt you to reboot, Exchange will tell you there is a pending reboot from a previous installation.
If you head over to Programs and Features in the control panel, make sure you only click on Symantec, so you get the additional menu — if you double-click it, you basically uninstall Symantec. Click on the change button on the top, and you will get to the Symantec install wizard. When you get to the section where the core files are, you need to expand “Network and Host Exploit Mitigation” and then click the drop-down arrow next to “Intrusion Prevention” and select the option “Entire feature will be unavailable,” as you can see below:
Click next and then follow the prompts to continue with the uninstallation of that feature. The process takes a few minutes to complete. Once you click Finish, you will see a popup from Symantec that the networking has been restored. By this time, you should be able to copy the Exchange 2016 and Exchange 2019 CU and CU security updates across the network. We will not dive into an Exchange 2016 or 2019 installation, but you can now mount your ISO file, launch an elevated command prompt, and either run the upgrade or perform a new installation.
You still have antivirus protection on the server while you perform the install. Once you are done with the cumulative update and security update for that CU, you can come back and turn that feature back on. Once the install completes, simply reboot the server, and IPS protection will be active again. Similarly, ESET mail security version 7.2 for Exchange 2016 or Exchange 2019 displayed the exact symptoms. The only way to fix that was to downgrade to version 7.1. But now that version 7.3 is out, upgrade ESET.
Passing the blame
Both vendors blame Microsoft, even though it appears that it’s their products causing the issue. Nevertheless, we have a workaround. With IPS disabled on your Exchange Server, if you are running Symantec or on ESET V7.2, you may notice strange errors in the event logs like the Exchange Servers constantly losing access to the domain controllers and cycling through them often. Your logs are full of warnings and errors because OWA has to keep restarting along with EWS. Then you get all the ASP.NET errors. A reboot fixes that for a few minutes before it all returns. Here is an example of this:
Intrusion Prevention causes these errors, and you may be wondering, “Why have I not whitelisted Exchange?” Well, I have whitelisted all the folder paths and processes, but IPS does not seem to look at it. IPS also does not write a signature to the event logs on the antivirus product, so you cannot grab a hash and whitelist it. If you reach out to the vendor and ask them to provide you with the hashes as Exchange should have the same for each version, they advise they cannot give it to you because it is not in the logs, and you need to open a call with Microsoft. You can see this becomes a ping-pong match.
Take this a step further: If you have IPS enabled, you cannot reseed a database copy on Exchange. It simply closes the connection after five minutes and hangs. The moment you disable IPS, you can reseed your Exchange database without error. Once the reseed has completed, the DAG runs fine, which does not make sense as it is still pulling/pushing log files over the network.
While I do not have an answer about the IPS whitelisting, I can tell you if you need to disable IPS temporarily to ensure that your systems run smoothly, you can. Just remember you do not have Intrusion Prevention protection enabled, but at least you have AV on the server and don’t end up with corruption.
Featured image: Shutterstock