Patch Management: More Important than Ever (Part 2)

If you would like to read the first part in this article series please go to Patch Management: More Important than Ever (Part 1).

Introduction

In Part 1 of this series, we looked at both the history and the current state of the security patching landscape, and began discussing your options for meeting the increasing challenge of managing the patching process in today’s mobile-centric, cloud-based, BYOD-driven networking environment.

Best Patch Testing Practices

Patch testing should be a routine process that follows a prescribed pattern, not a haphazard task. Your team should lay out written procedures for testing and it should be done in a test lab that is as similar as possible to the configuration of your production network.

Some organizations completely isolate the test network from the production network. Others use a small group of production machines on which to test the patches. The sophistication of the testing environment that you’re able to construct may, of course, be subject to time, personnel and budgetary constraints. Virtualization can help to cut the costs of setting up a test environment; this will allow you to use one physical machine to duplicate many of your production machines.

Before the updates are applied to the test lab machines, though, the integrity of the patches should be checked to ensure that they are valid updates from the vendor and haven’t been deliberately or unintentionally modified. This is the patch validation process and can be done by verifying digital signatures or checksums.

Thorough testing involves applying the patches and rebooting the machines – whether or not a reboot is deemed necessary for the changes to take place. This is done to ensure that the operating system’s boot process hasn’t been affected by the patch. It’s no fun to apply a patch and think all is well, and then get a nasty surprise days or weeks later the next time the system restarts.

A normal reboot doesn’t necessarily mean you’re home free, however. The infamous “blue screen of death” is the most severe but by no means the only impact that a bad patch can have. You should have a set of test scripts that will help you determine whether the operating system and your applications continue to operate correctly after the patch application. A security patch can affect many different parts of the system so you need to test it in different scenarios that mirror the most commonly performed system tasks.

A change management solution should be a part of your test environment, as this will automate the detection and pinpointing of changes that the patch may have made to your systems, such as modification of permissions and access controls. Patches may also disable or enable services, make changes to the registry or changes to system or application code that can cause OS features or application functions to fail completely or to behave abnormally.

If the patches cause no problems in the test environment, then you can prepare to roll them out to production machines. Depending on the severity of the vulnerability, the impact (based on the mission-criticality and sensitivity of your machines) and the likelihood of exploit (based on your machines’ level of exposure and whether there are exploits already in the wild), you may deploy the patches across your entire network at the same time, or you may roll them out in phases.

Doing it in phases can be safer, and can function as an extension of the patch testing process. You would apply the patches to less critical systems first and then evaluate the impact on those machines for any negative consequences before rolling out to your more important servers. If at any point in the testing process, including after part of a phased rollout, you shouldn’t deploy the patches to any additional machines until the problems are resolved.

Because something can always go wrong even after thorough testing, part of the testing process should include ensuring that the patches can be uninstalled and the systems restored to their former state without problems. Be sure that you have a contingency plan if you need to back out of the patch as well as a recovery plan if important systems are affected.

Patch deployment options

The deployment phase is where the patches are installed on the machines on your production network, and today’s IT pros can be thankful that this is no longer the tedious manual task that it was in the early days of computing. There are numerous tools that can be used to automate the deployment process – but therein lies the problem: you have so many options that it can be difficult to decide which approach to take.

Some organizations still use their own custom scripts to apply patches, but an entire industry has grown up around patch deployment and there are now many commercial products that can make the whole process much easier. Some support only a single platform (e.g., Windows) while others can manage the patches for different platforms in a hybrid network environment.

Since Windows Server 2003, Microsoft’s server operating systems have included built-in update management functionality in the form of Windows Server Update Service (WSUS), which was first called Software Update Services (SUS) in the earlier versions. It has expanded its scope from being limited to updates for Windows only to being able to update other Microsoft products such as Office. WSUS is now installed as a server role on Windows Server 2012/2012 R2.

The WSUS server connects to the Microsoft Update Service to get current update information; it downloads the updates and then distributes them to the computers on your network according to your specifications. In a large organization, WSUS servers can function in a hierarchical manner, with an upstream WSUS server distributing the updates to other WSUS servers on the network, which in turn distribute the updates to the local servers and clients.

The big advantage of WSUS is that it comes with Windows Server, so you don’t have to buy additional software. Microsoft’s System Center Configuration Manager can be used for patch management along with asset discovery. SCCM is well integrated with WSUS and Windows Update Agent. The patching process can be automated with, for example, different patching schedules for different departments or other groups within the company and it’s easy to exclude some machines from the patch deployment and also customize restart behavior for different groups of machines.

Whereas WSUS provides good, basic update control for Microsoft software and SCCM gives you more options and flexibility, there are much more sophisticated patch management solutions that are offered by third party vendors, and many of these also integrate with WSUS and/or System Center.

These systems can enable you to more easily manage patching in a centralized way for hybrid networks that have non-Microsoft operating systems running, as well as easier patching of third party application software on Windows. Although you can integrate System Center Updates Publisher (SCUP) with SCCM, it takes a great deal of administrative overhead (read: time and effort) to apply third party application updates, especially when the application vendors don’t provide SCUP-compatible CAB files for their updates. Third party products can also provide you with such extras as real time patch failure reporting and notification and compliance scanning.

There is a plethora of patch management solutions available and it’s beyond the scope of this article to attempt to review or recommend one, since the best solution for you depends on your organization’s specific needs and your network’s configuration. Some of the best available products include GFI’s LanGuard, Lumension Patch & Remediation, SolarWindows Patch Manager and Secunia CSI (Corporate Software Inspector).

In selecting the best patch management solution for your organization, you’ll want to think about such issues as whether to deploy all patches from a centralized location or have multiple update servers located at different places on your network to serve particular groups or departments. This will depend on how large your organization is and how it’s structured. In either case, a good patch management solution will be able to validate the patches.

Another consideration is whether to use a patch management solution that uses client agents or one that is agentless. There are advantages and disadvantages either way. The agent-based systems require that you install software on each of the clients, but they work better for updating machines that are in a DMZ and they can use less bandwidth and spread the processing load between client and server. Agentless solutions can discover computers that are new to the network without installing agents on those clients and reduce the administrative overhead of installing agents. You can combine the two solutions, as well.

An important criteria for selecting a patch management system is to be sure that it can evaluate whether the patches themselves have been updated or revoked or replaced. It should also be able to take into account whether patches need to be applied in a specific order. Finally, it should have good reporting capabilities. This is even more important if your organization is part of a regulated industry that falls under compliance mandates, since you will have to be able to document that your machines have all the correct security updates applied.

Summary

Patch management isn’t something that most IT pros get excited about, but it’s a good idea to reassess your patching strategy and tools every now and then, to ensure that your updating process is as efficient as possible and to save yourself unnecessary work. There are many options available today for automating the tasks involved in keeping your systems updated and secure, so go out there and check them out.

If you would like to read the first part in this article series please go to Patch Management: More Important than Ever (Part 1).

Leave a Comment

Your email address will not be published.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top