A sysadmin’s patch management conundrum: Patch now or wait?

You’ve just received a bulletin from your vendor about a newly found security vulnerability. A patch is also available. Should you apply it now or wait? You should wait, of course. Let’s explore why. In the IT security community, the general mantra is when it comes to patch management, “patch early, patch often.” On the face of it, this looks like good sense. After all, the window of time these days between a critical vulnerability being discovered and an exploit for it appearing in the wild is decreasing, and often almost zero or even in the negative. So as long as you’ve had a chance to perform some testing on a newly released patch, it seems logical to go ahead and apply it quickly in your production environment.

patching

Patch management/patch mismanagement: Patches can break things

Here’s where things go wrong with patch management. Because patches often break things. I can’t count how often Microsoft has released a patch to address some issue, and the patch itself has then caused additional problems. It’s like having a mechanic fix the steering on your car and ending up having problems with the brakes because of something the mechanic did. As a recent example of this happening, look at what happened with the CVE-2020-17049 Kerberos KDC Security Feature Bypass Vulnerability. This vulnerability involved a security feature bypass that existed in the way the Key Distribution Center (KDC) service for Active Directory determines whether a service ticket can be used for delegation via Kerberos Constrained Delegation (KCD). For a malicious actor to exploit this vulnerability, a compromised service configured to use KCD could tamper with a service ticket that is invalid for delegation to force the KDC to accept it. The security advisory published by the Microsoft Security Resource Center (MSRC) then addressed this vulnerability with guidance on changing how the KDC validates service tickets with a fix using the following registry value:

HKLM\System\CurrentControlSet\Services\Kdc\PerformTicketSignature

Sounds pretty straightforward, doesn’t it? Vulnerability is discovered, vulnerability is fixed.

Then, however, they discover that their fix could cause problems for certain environments, such as having Kerberos service tickets and ticket-granting tickets (TGT) not being able to be renewed for non-Windows Kerberos clients, plus some other nasty difficult-to-troubleshoot stuff. So, they release another patch for this and publish a support article concerning it. And so it goes. Maybe it would have been better to wait until Microsoft patched the patch (or fixed the fix or whatever) before applying/performing it in production.

Patching guidance can change

There’s another reason why it’s generally better to wait than patch or fix right away when a vulnerability is announced. This is because vendor guidance for a patch or fix may change. Once again, let’s use the CVE-2020-17049 as an example. On Nov. 10 of last year, when Microsoft first released their security advisory for this vulnerability, they included the following guidance:

FAQ

Are there any additional steps I need to take during deployment of this update?

Yes, for complex domain environments a registry key has been provided to allow for deployment across domains before fully enabling the fix. In a complex forest, where Kerberos tickets may travel across multiple domains, we recommend the following steps:

  1. Set the registry key to 0 (disabled).
  2. Complete the deployment to all DCs (and Read-Only DCs) in your forest.
  3. When deployment is complete, set the registry key to 1. A later release will remove this registry key and make ticket signatures required.

The initial guidance from the MSRC then detailed what steps you need to perform using the PerformTicketSignature registry value.

Two days later, however, if you looked at the advisory again, you would see that the above guidance has been modified and now reads like this:

FAQ

Are there any additional steps I need to take during deployment of this update?

In a simple environment with a single domain, no additional action is required.

In environments with multiple domains, a registry key has been provided to allow for deployment of the patch to all domain controllers (DCs) before enabling the new behavior. In these complex environments, we recommend the following steps…

What if you had only a single domain and applied the fix immediately? Would it have had any negative effect? Would your actions need to be undone somehow? There’s nothing from Microsoft that answers this, but it raises the question of whether it might not be better to wait for proper guidance before implementing a patch or fix.

Fast forward to today. Now, if you look for guidance at Microsoft’s advisory on this matter, you’ll find something completely different:

FAQ

Do I need to take further steps to be protected from this vulnerability?

Yes. Customers who have already installed the Nov. 10, 2020, security updates need to install the Dec. 8, 2020, updates. The Dec. 8, 2020, security updates include fixes for all known issues originally introduced by the Nov. 10, 2020 release of CVE-2020-17049. This update also adds support for Windows Server 2008 SP2 and Windows Server 2008 R2.

For more information and further steps to enable full protection on domain controller servers see Managing deployment of Kerberos S4U changes for CVE-2020-17049.

What if you haven’t installed the Dec. 8 updates for some different reason, say because of an application compatibility issue? Or perhaps you haven’t even installed the Nov. 10 updates. What should you do now? Read the other article that’s linked here, I imagine. And there’s quite a lot to digest there.

What’s worse, in my opinion, is that the advisory now uses the “new and improved” structure and presentation, which almost all of my colleagues who work in IT administration in Windows Server environments tell me they detest. Mainly because the Executive Summary section has been removed from these advisories, making it more difficult for those reading it to digest it properly.

My guess is that Microsoft removed this summary because they struggle themselves to know how to properly address some vulnerabilities. Hence, they don’t want to offer a quick summary of the matter in case they discover they’re wrong afterward. In other words, they’re fixing things on the fly. Or perhaps flying by the seat of their pants. You can choose the metaphor you like best to describe this.

So, when it comes to patch management: patch or wait?

Wait. Yeah.

Featured image: Shutterstock

5 thoughts on “A sysadmin’s patch management conundrum: Patch now or wait?”

  1. That’s the sad consequence of Microsoft way of doing things. They keep adding “features”, but always had a “security as an afterthought” approach to programming. They also always liked doing things “the incompatible way” (vendor lock-in), instead of following standards.
    I use linux at home, and never had second thoughts about installing patches. Ok, occasionally some scary stuff appears during patching, like grub configuration messages, but that’s pretty much all.

  2. Hmm, I don’t think you can really say:
    “So, when it comes to patch management: patch or wait?
    Wait. Yeah.”

    I get where you are coming from in this article, been patching systems for 20+years now, but it’s not that clear cut to wait or patch now. It’s a “Depends” scenario – any vulns that are serious and public exploited should be patched now especially on systems at high risk. Other patches can wait for testing and validation or just wait to see what is reported.

Leave a Comment

Your email address will not be published.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top