If you would like to read the next part of this article series please go to Patch Management: More Important than Ever (Part 2).
Introduction
the most essential elements of network security is keeping all of the software on the network up to date. That might seem like old news, but in this era of exploit kits that make it possible for even the non-technically savvy to launch sophisticated attacks, patching takes on a new importance. In this article, we’ll look at your options for meeting the increasing challenge of managing the patching process in today’s networking environment, where the BYOD trend has brought us a plethora of different operating systems and applications running on different types of hardware that aren’t owned and under the complete control of your organization’s IT department.
Evolution of patch management
Not so long ago, software vendors used to issue fixes on floppy disks and then CDs – which meant shipping costs and a relatively long amount of time between the creation of the patch and the ability of customers to apply it to their own systems. The Internet changed all that, very much for the better. Global connectivity meant customers could download patches as soon as vendors made them available.
Even though this level of convenience made it far easier to update systems, it wasn’t enough. Many people still failed to apply the patches in a timely manner. When most fixes were designed to address reliability and stability issues, this only created problems for the procrastinators themselves. But as the incidence of attacks and malware that take advantage of software flaws began to proliferate, failure to patch began to impact not just the individual machines but entire networks.
The security ramifications changed the rules of the patching game forever. Applying fixes was no longer something to do whenever you got around to it. Installing updates became a top priority for IT professionals. At the same time, getting patches written and released as quickly after the discovery of a vulnerability became a top priority for the software vendors.
The complexity conundrum
This was exacerbated by a rising number of security researchers who adopted a philosophy of exposing information about vulnerabilities to the public before vendors created patches, thus creating Zero Day situations whereby that information is available for potential attackers to exploit. The merits of that approach can be (and have been) debated endlessly, but the practical effect is that it makes it more important than ever to get the patches installed as soon as possible after they are released.
“As soon as possible” is the key phrase here, and that doesn’t necessarily always mean “immediately after they are released.” While patching has evolved to the point where automatic updating processes will do it all for you, and whereas this might be the best practice for consumers, it’s not always prudent to just “set it and forget it” in a corporate environment where downtime from a botched patch could have a high dollar impact on the company’s bottom line.
Unfortunately, patching today is not only more important than ever; it’s also more complex than ever. In the “olden” days, most networks consisted of a number of servers (running Windows Server in a “Windows shop” or some variety of Linux/UNIX in a “*NIX shop”), along with a number of desktop client machines, usually Windows clients. This uniformity made it relatively easy to roll out patches across the organization.
Today, uniformity is the exception rather than the rule. Most corporate networks run a mix of server types; you might have a Windows domain with Linux-based web servers running Apache. On the client side, it gets even more complicated. Companies have opened up their networks to employees’ personally-owned devices of all kinds. Telecommuters have their own desktop machines at home, which might run one of several versions of Windows, Mac OS X, or one of countless distros of Linux. Users who work on the company premises bring in their own laptops. Employees no longer access company resources only from these computers, but may also work from their tablets or phones, running one of a variety of mobile operating system versions. Ensuring that all of these are properly patched is a major headache for IT.
The good news is that designing a well-thought-out patching strategy that works best for your organization and deploying the proper policies and tools can help to take some of the pain out of patching.
The “quick fix” dilemma
The need to patch as quickly as possible due to the dangers posed by Zero Day threats directly conflicts with another important tenet of updating: the need to carefully test patches in a controlled environment before rolling them out on your production network. This second need is also more important than ever, because the haste with which vendors are scrambling to devise and issue patches also means there is a greater chance than ever that they’ll be released before the vendor has time to test them as thoroughly as is desirable – and thus more of a chance that there will be undiscovered conflicts that can result in problems when applied to some configurations.
Due to the nature of software, it’s never possible for a vendor to cover every possible configuration in testing. That’s why best practices have long included in-house testing on systems designed to emulate your production systems (in terms of hardware, operating system versions and settings, applications installed and so forth). The trick is to protect your systems as much as possible from vulnerabilities, without putting them at a different kind of risk from untested patches.
The first step toward that objective is to evaluate each patch individually and in effect perform a risk assessment for each vulnerability. That means looking at the following factors for every patch:
- What the probabilities are that your machines will come under attack from an exploit of a particular vulnerability, taking into consideration the general exposure of the machines to the Internet.
- Whether any exploits have been reported in the wild.
- How mission-critical the data on particular machines is and what the business impact would be if that server were brought down by a problem patch.
- How sensitive the data on particular machines is and what the business impact would be if that data were exposed or lost.
Patching priorities will be different for different organizations and even for different machines within an organization, based on these considerations.
Do your homework
IT pros are already busy enough without having extra chores piled on top of it, but when it comes to patching, a few minutes spent doing a little research can potentially save you hours of frustration resulting from a patch that causes major problems. In order to make the evaluations suggested above, you need information about both the vulnerabilities that are being patched and the patches themselves.
That means you need someone on your security team who has a deep understanding of operating system, application and protocol vulnerabilities and exploits, how they work and how to intelligently estimate the level of risk that each poses.
There is a wealth of information available on the web regarding the latest vulnerabilities but it’s important to be able to vet the sources for their reliability and accuracy before blindly accepting their advice. Some sites oversensationalize their reports to draw in more readers and others may downplay the risk to avoid criticizing the software vendors that support them through advertising and other partnership relationships.
Within a few days after a major set of patches is released, if there are widespread problems associated with the patches, the news will usually begin to emerge first in the tech forums and then will be picked up by the tech press. Scanning the forums for such information can tip you off early that some patches might be best to avoid or at least to set aside until you get more information. It’s sometimes possible to save yourself a great deal of trouble by simply waiting a few days. The really bad patches are often yanked by the vendor and replaced with a “fixed fix” within a short time.
Of course, even if most systems don’t have any problems in the wake of the patch application, there is always a chance that your particular configurations, settings and applications could still conflict with a patch. That’s why it’s important to have a patch testing lab in place where you can try out patches – especially those reported to have some known problems but not so widespread or so severe that the vendor recalls them – before you install them on your mission-critical servers.
Summary
In this, Part 1 of a two-part series, we discussed the ever-growing importance of security patching and some basic tenets of best management best practices. In Part 2, we will delve more deeply into patch testing and the many patch deployment options that are currently available, including both on-premises and cloud-based solutions for automating the patch management process.
If you would like to read the next part of this article series please go to Patch Management: More Important than Ever (Part 2).
