To patch or not to patch? In an ideal world, that wouldn’t be the question. Of course every organization should apply the security updates for their operating systems and critical applications, and they should do it as soon as possible after those updates are released. Once the vulnerabilities have been disclosed, it’s only a matter of time – and sometimes not much time at all – before attackers use that information to devise exploits. For whatever length of time goes by between a patch release and when you install it, your systems will be in a zero day state.
Why, then, are we hearing that more and more companies are delaying the installation of critical security patches? Following a number of recent high profile “patch fiascos,” IT departments (and individuals) have become wary of allowing automatic installation of security (and other) updates, lest the “fix” may prove to cause more trouble than it’s worth.
Over this last year, we’ve seen numerous instances where Microsoft’s regular “Patch Tuesdays” have led to “Recall Thursdays,” with various patches breaking Office, affecting the functionality of Windows and even resulting in complete system crashes and the dreaded Blue Screen of Death. It’s not just Microsoft updates that are causing problems, either. Mobile technology is at the forefront of the enterprise workplace today, and after Apple’s recent release of iOS 8 for their iPhones and iPads, many users reported problems including fast battery drain, wi-fi signal that’s slow and/or drops out, sound not working, random reboots and more.
The dilemma is that if you try to avoid these problems by not installing the updates, you make the computer or device (and consequently perhaps your entire network) vulnerable to exploits and attacks. Some of the troublesome Microsoft patches were designed to address critical vulnerabilities, and iOS 8 includes over fifty important or critical security fixes.
Of course, this update phobia is nothing new. Consider the number of people (almost 13 percent of the market as of August 2014) who are still using Windows XP – a thirteen year old operating system that is no longer supported by Microsoft. In some cases it’s due to simple laziness, cost, or a distaste for change, but I’ve heard many XP holdouts say that they didn’t want to buy a new computer and were afraid upgrading the ones they have would “break things.”
Microsoft’s own Security Patches Best Practices documentation on the TechNet web site contains the statement that “the risk of implementing the service pack, hotfix and security patch should always be less than the risk of not implementing it” and goes on to say “You should never be worse off by implementing a service pack, hotfix and security patch. If you are unsure, then take steps to ensure that there is no doubt when moving them to production systems.” Unfortunately, it seems many of their customers are unsure these days. So what are the steps you need to take to ensure there is no doubt?
Timing is everything
Obviously sticking your head in the sand and ignoring all updates out of fear that one will crater your systems isn’t the wisest course of action. Most large enterprises with big IT departments have protocols in place for testing of patches before deploying them to the entire network.
However, smaller businesses don’t always have the luxury of sufficient personnel and hardware to set up and maintain a test lab environment. That’s why many have begun to take a “wait and see” approach, delaying patch installation for a week or two in order to let someone else be the “guinea pig.” Then, if no major problems emerge in the tech press after that time, they’ll go ahead and roll out the patches to their own machines.
Another advantage of waiting is that even if the software vendor promptly issues new patches, or “fixes the fixes” after problems are identified, it can create a major pain point for those who then have to uninstall the original patch before they can install the new one. If you simply wait, the problematic patch may be removed from availability and replaced with one that works as intended, and that you can install fresh without having to go through the uninstallation process.
Tips for safer patching
Even without an elaborate testing setup, there are things you can do to lower the risk associated with patching. Some require a bit of time and effort, but likely not as much as dealing with a bad patch (or with the consequences of failing to patch at all) would require.
The first and foremost bit of advice is a variation on the old RTFM standard: RTFD or Read the Fabulous Documentation. Some software vendors provide more documentation than others, but most put out at least minimal explanations of what the vulnerabilities are that are being addressed, what specific software components are involved, and what any prerequisites are for installing the patch.
Often problems could have been avoided if the user or IT admin had simply read all of the documentation before applying the updates. Some updates may require that other software be installed (or uninstalled) before you apply the fix. In some cases, you might need to enable or disable specific OS or application features. In many cases, updates need to be applied in a particular order.
Look before you leap
The official documentation for the update (for example, in the case of Microsoft security updates, the associated Security Bulletin and KB article) sometimes don’t tell the whole story. If the update has been released for a few days, even if problems have already been reported, software vendors can sometimes be slow to acknowledge the problems, much less fix them.
That’s why it pays to do a web search and scan the community forums to get early reports on any problems that others might be experiencing when they apply the patches. Forum discussions might also turn up trends; it may be only users with specific configurations or with particular settings enabled or with a certain third party program installed who are being affected by the problems. This gives you more information on which to base your own assessment of whether it’s safe to deploy the patch to some or all of your own machines.
Once you have all the information that you can gather about the update(s), you can intelligently consider whether the risk inherent in installing is worth the benefit to be derived from it. That means looking at several different factors about the particular update and about your machines, network and business practices.
- What does the patch fix? Is it a security update or is it a non-security update that fixes reliability or performance issues or other bugs, or that adds features and functionality?
- If it’s a non-security update, are your systems affected by the reliability or performance issues that it addresses? Software vendors often recommend that hotfixes be applied only if your machines are actually experiencing the issues. Remember the old advantage “If it’s not broke, don’t fix it.”
- If it’s an update designed to add features and functionality, are these features that you need or want? Will you use the new features? If not, why take the risk of installing the update?
- If it’s a security update, what is the particular vulnerability (or vulnerabilities) that it addresses? What is the severity level of the vulnerability? What is your company’s exposure to the risk? That is, what are the scenarios under which the vulnerabilities could be exploited and do those scenarios apply to your users? If you have user workstations and applications locked down so that they can’t, for example, open email attachments, and the only way the exploit can be distributed is through an email attachment, your exposure is minimal and you might not consider it urgent to apply the patch – especially if it’s one that has the potential to cause functionality problems.
Have an exit strategy
There’s an old bit of sage advice that I try to follow in all aspects of my life: “Hope for the best, but prepare for the worst.” It’s a good adage to remember when applying software patches.
Lo, those many years ago when I was a law enforcement officer, we had it drilled into our heads during the police academy that a basic tenet of best officer safety practices is to always be completely aware of your surroundings, and know where all the exits are. This is also stressed in civilian safety training; one of the most important things the flight attendant tells you in the safety briefing is to look around and note where the nearest exit is.
In this context, having an exit strategy refers to having a plan for rolling back the updates if they do cause problems and restoring your machines to their pre-patch functionality, with a minimal amount of down time. That involves having good, current backups and having a recovery process already in place that has been tested and works.
Don’t put all your eggs in one patch basket
Even after you’ve done your homework regarding the patches, even after you’ve done a cost/benefits analysis and determined that the risk of not updating outweighs the possibility of patch-induced problems, even when you’ve formulated a good exit strategy, it still pays to hedge your bets. Even if you aren’t able to do a full test-lab experiment – or even if you have tested in the lab and found no problems there – the safest way to proceed is to apply the patches first to only a small sampling of non-critical production machines.
This gives you the opportunity to evaluate how the patch behaves in your real environment and catch any problems that didn’t crop up earlier. And even when all goes well there, it’s a good idea not to become complacent until the patches have been deployed to all machines that need updating. You can continue the gradual rollout over a matter of several days – and be sure to prepare your users for the possibility of problems, even those problems that are attributable solely to functionality changes and user unfamiliarity with the new features or new ways the software behaves.
Yet another valuable lesson that police officers in training learn is that “there is no routine traffic stop.” Even though most of the time, pulling over a vehicle for a traffic violation goes smoothly and presents no danger, many officers have been killed when they walked up to a car to issue a ticket for a minor offense such as speeding. It’s vital to keep in mind that there’s always a possibility that bad things will happen.
Software patches aren’t going to result in physical injury or death, but the same type of strategic thinking and awareness are still useful, and can definitely reduce the amount of damage done (in the form of extra work, lost productivity and resultant monetary cost to the company) in case this one turns out to be anything but just another routine patch job.