On the most recent Patch Tuesday, Microsoft addressed 55 security vulnerabilities across its Windows operating systems and other software. Six of these software updates were marked as Critical while the other 49 were marked Important. When I saw the notification my first thought was, “Should I apply all these patches now or should I wait a few days to see whether any of them might cause problems for the network I manage?”
This problem is one that frequently arises for administrators of environments that have Windows client and server systems. One can distill the matter into a simple question: How fast should I apply software updates when my vendor releases them?
The answer of course, like most things in life, is that it depends. In this article I’ll explore some considerations when trying to make the right decision.
Critical doesn’t mean you should panic
Panicking doesn’t solve anything when it comes to managing IT. So when Microsoft says that one or more of their patches address critical vulnerabilities in their products, stay calm and evaluate whether your environment is really at risk. A good place to start is to look at the CVE for that patch. How could the vulnerability affect your environment? Where might it first arise and/or do the worst damage?
A good way to prepare for these evaluations is to divide your IT environment into three priority levels: the most critical systems and infrastructure, the parts that are essential but not highly critical, and the remainder that are not as critical should a problem arise. Then review the CVE for a Critical software update Microsoft just released and give highest priority to examining whether it could affect your most critical systems.
Always test updates even if they address zero day vulnerabilities
Just because a vulnerability is identified as a zero day doesn’t mean you should rush to deploy it immediately in your environment. You still need to test any updates for this vulnerability before you roll it out into your production network.
I know testing isn’t much fun. It’s work. It necessitates the expense of setting up (and maintaining!) a testbed environment that mostly mirrors your production environment. But partitioning your network into most, some, and not very critical parts can be a big help. You must — really must — test software updates that apply to the most critical systems in your infrastructure. Typically this means testing some of your servers (virtual or physical), regardless of whether they’re hosted on premises or in the cloud. Note that I said “some” servers. There may be server workloads whose functions are not that critical to supporting your core business processes.
On the other end of the spectrum are your workstations. Patching workstations may not be major issue. In fact, you may not even want to test workstations if it is easy to simply wipe and reinstall a PC if a patch creates a problem.
For the systems that are somewhat critical, a good approach might be to wait and see what effect the patches have on your most critical systems after you apply them. If, after a couple of days, your critical systems seem to be running well, go ahead and deploy the updates to your less critical but still essential systems.
On the other hand don’t forget backups
Sometimes zero day really does mean some of your systems are in imminent danger of being compromised. In this case, act immediately to apply the patch that the vendor has released to address the vulnerability. An example of this would be the Microsoft Exchange zero day exploits identified in March of 2021. It was discovered that these exploits were already happening in the wild and being used to attack on premises versions of Exchange. As news about this rapidly spread, many administrators who had Exchange in their environments rushed to apply the mitigation steps promulgated by the Microsoft Security Response Center (MSRC). Administrators then quickly applied the patches Microsoft released for the vulnerability.
Keep in mind that when a zero day exploit has been found, your critical systems may have already been compromised! This realization is important as it explains the relationship between patch management and business continuity. In other words, it’s no good patching systems that are already broken. You need to restore them from backup first. So having a good approach to backing up your critical systems is equally important to keeping them patched against critical exploits.
Understand your tolerance for risk
As the network administrator, you need to feel comfortable about how much risk you can tolerate for your environment. Even more importantly, as you build out or upgrade your network, you should try to do so in a way that minimizes future risk as much as possible. Risk tolerant architecture like this takes some time to develop, and there’s no magic way of achieving it. There will always be some risks you can’t eliminate. But each time you need to add a new server, deploy new software, or built out a new network, ask yourself questions like:
- “Will doing this add any new risks to my environment?”
- “How can I minimize these new risks?”
- “Will there be any increase in risk if I don’t make this change to my environment?”
