Microsoft releases Cumulative Updates and Security Updates for Exchange Server (2016 and 2019) every three months. Each of these Cumulative Updates and Security Updates comes with several fixes and security fixes. Microsoft advises that you install the Security Update for the Cumulative Update you are installing. You should first upgrade an Exchange 2016 or Exchange 2019 Server to the latest Cumulative and Security Updates in a lab before applying it to your production servers, although I realize that for many companies this is not an option. You may be asking a question that many others are asking: “Why does the update work fine in my lab but when I apply it to my production servers I end up with issues?” This is a question that I had with Exchange 2016 CU21 Security Update, the latest CU and SU released by Microsoft in June. Both my labs were upgraded to Cumulative Update 21 for Exchange 2016 and Cumulative Update 10 for Exchange 2019 without error, and there were no errors after a reboot. The Security Update for each of these Cumulative Updates was installed and monitored for a week with rigorous testing, which did not show any symptoms or issues.
Exchange 2016 CU goes smoothly — or so it seems
I started with a client upgrade that was requesting the upgrade to Exchange 2016 CU21 and the Security Update for that CU as they wanted to make sure all angles were covered from a patching perspective. The upgrade of Exchange 2016 CU21 was fine — no errors were encountered — and after the reboot, the log files were clean. The Security Update was then applied, and it completed successfully. (It does take some time to install.) The server was rebooted once more to complete the installation, and upon checking the log files again, no errors were reported. The next server was upgraded, and it too did not give any errors — both were humming along.
Help desk starts getting calls
After about 20 minutes, the help desk started receiving calls that OWA was not logging them in or was logging them out and they could not log back in. I checked the event logs on the server for any issues, but no errors were reported. I started checking other forums, and people were posting that the Security Update for Exchange 2016 CU21 seems to break OWA and the ECP, but Outlook clients work fine. In this regard, all further upgrades were halted until a fix is made available for this.
What is strange is that another client we upgraded to CU21 for Exchange 2016 along with the Security Update did not show any symptoms, and OWA and the ECP were working fine. It was mentioned the Exchange Auth Certificate had to be renewed to fix the issue, but one environment had new servers and the other older servers from an installation perspective, so could it be the new one was fine because its install date was newer? I don’t know the answer. All that was done to solve the error with OWA and the ECP was to take the servers off the load balancer for now.
My advice: Hold off on CU21 for Exchange 2016
My advice: I would upgrade to CU20 for Exchange 2016 and its corresponding Security Update until we know why the Security Update for CU21 is causing issues.
Now, let’s shift gears and talk about the part of your Active Directory environment needing a schema upgrade. It is important to always read the notes on a Cumulative Update to understand what is required and what is fixed from a previous build. Microsoft made it known that when you upgrade to CU21 for Exchange 2016, you have to perform a schema upgrade. With that being said, you need to have RSAT Tools installed on a server that can run this and there is a two-step approach. One is to upgrade the schema. The second is to Prepare AD. If you have a large environment, running these commands may complete successfully, but you need to wait for AD replication to take place. I generally do these two steps the day before to give the environment time to sync and settle.
The two commands, which I have covered in other articles here at TechGenix, are listed below as a refresher and for those new to Exchange 2016 or Exchange 2019:
Setup /PrepareSchema /IAcceptExchangeServerLicenceTerms Setup /PrepareAD /IAcceptExchangeServerLicenseTerms
When you have completed these two commands and are comfortable moving forward, you can run the upgrade of CU21 for Exchange 2016 or CU10 for Exchange 2019. I know I have said to upgrade to CU20, but you can test a server if you want to so you can see if you run into the OWA errors afterward. As I said, two different environments did not show the same symptoms.
Exchange 2016: The clock is ticking
This brings me to the part where it is time for you to look at moving to Exchange 2019 if your company does not want to move to Microsoft 365. Exchange 2019 with hybrid is still receiving updates. You are encouraged to start your migrations to the platform of choice. If you are sticking with Exchange 2016, just remember that it is already on extended support. End of life for Exchange Server 2016 is scheduled for Oct. 14, 2025.
Featured image: Shutterstock
Eric, Can I just add my experience to your log?
I upgraded a client’s ex 2016 to CU10 on the weekend and ran into the issue mentioned above. The symptom was that after the update, OWA and ECP would not launch and were presenting a certificate error. The certificate being complained about was the oauth one, a self generated certificate, and Internet searches indicated that this causes a problem if it is close to expiry.
The steps to replacing this cert were simple, and there are blog posts all over the place on how to do this, but many mention that it took an hour for the cert to become active.
I found it took about 15 hours (lucky it was over a weekend!) because I am in the +8 time zone. i.e. when the cert is generated it seems to be generating for a -7 time zone (Seattle).
So it is possible that ECP and OWA will break when you run this patch (maybe look at updating the cert before starting the CU?) but the recovery is simple, however time consuming.
oops meant to say cu21 🙂
Have you tried running the CAS update powershell script?
& ‘C:\Program Files\Microsoft\Exchange Server\V15\Bin\UpdateCas.PS1’
or sometimes if just /ecp is broken, it’s an issue where the apppool has a bad binsearchpath
https://www.msnoob.com/exchange-ecp-and-owa-error-after-update.html
This happens often when I apply cumulative updates, so we’ve added a manual test before adding the CAS back to our load balancer, to locally check both before hand. Perhaps it’s not your issue, but this has been the fix for me a number of times.
the cert for my server is going to expire soon, i’m at CU19
is it better to prep to renew the cert, run the CU21 update
give it time and then see if it fails or not then apply the new cert?
Perhaps a bit late, but I experienced the same issue as you when I upgraded to CU20 and then installed the July 2021 security update. After much troubleshooting I was informed that there is a bug in the security update that is only manifested when you are using SSL offloading on a load balancer in front of the Client Access Servers. Disabling the SSL requirement for the ECP and OWA virtual directories (both on the default website and the Backend website) and restarting IIS should resolve the issue. I was also informed to make sure that all servers participating in the load balanced pool need to be running the same CU and SU version, otherwise the ECP and OWA logins will not be processed correctly.