Have Microsoft patches reached a painful tipping point?

When it comes to patching the Windows Server platform, various versions of Microsoft Windows, Microsoft Office, and other Microsoft applications, it almost seems to me like Microsoft is broken. In the old days, which in reality were not very long ago at all, those of us who work in the field managing or supporting Microsoft products felt confident that Microsoft thoroughly tested hotfixes, rollups, security updates, and other software patches before releasing them to customers. Sure, there were occasional times when something unexpected happened and a patch that fixed something ended up breaking something else either in Microsoft’s product line or for some third-party software vendor. But by and large, it seemed that patches got tested, tested, and tested again before they were let out of the barn door into the wild. But these days many of us wonder whether Microsoft — in its eagerness to push its customers away from on-premises deployments into their ever-expanding stable of cloud services like Azure and Office 365 — may have simply lost interest in investing the time, energy, and budget needed to ensure their development teams produce software updates that don’t cause customers more headaches than they solve.

In short, it almost feels like patching (including the development, testing, communications, and release of patches) at Microsoft is broken, or is in danger of soon reaching that state. And Patch Tuesday, that second Tuesday of each month when Microsoft releases security updates to fix bugs they’ve identified in their software products, has become as dreaded for many admins as Friday the 13th was for the Knights Templar when Philip IV arrested hundreds of them on Oct. 13, 1307. Below I’ve shared some examples of the kinds of things I’ve been hearing in recent months from frustrating admins who work with Microsoft products and patch Windows-based infrastructures ranging from a handful of computers to tens of thousands of deployed systems. In a future article here on TechGenix, I’ll discuss some mitigation strategies you can follow and best practices you can adhere to reduce the potential risk of a nightmare happening when you patch your organization’s Windows clients and servers.

Stories from the field

Most of the below incidents have happened within the last three months, and it seems to many of us that patch development/management has been rapidly going downhill at Microsoft during this time period. To protect the identity of those involved I’ve fictionalized these incidents by altering personal details, but they’re all true stories in their technical essentials:

• Installed the latest Security Monthly Quality Rollup for Windows 7 for x64-based Systems on my Dell Optiplex and it froze the system. This caused me much grief because it was my main work machine and it took me an hour to make it bootable again from a restore point. I’m sick of Microsoft making us their live testers for their software updates!!
• Really? To address the Speculative Store Bypass (SSB) vulnerability in Windows with regard to the newly identified Spectre/Meltdown variants on Intel systems, I need to go through the steps of applying Windows updates that provide support for SSBD, then manually download and apply hardware/microcode updates from my device OEM, then manually change a registry setting to turn on Speculative Store Bypass Disable (SSBD) on the CPU, then run a Windows PowerShell script, then… C’mon! How can we be expected to do this manually for all the PCs we manage on our network?
• From a Microsoft Support article for a rollup for Microsoft Exchange: “When you try to manually install this security update in ‘normal mode’ (not running the update as an administrator) and by double-clicking the update file (.msp), some files are not correctly updated. When this issue occurs, you do not receive an error message or any indication that the security update is not correctly installed… To avoid this issue, run the security update in elevated mode, as an administrator. To do this, right-click the update file, and then click Run as administrator.” Thanks again, Microsoft, because you can’t right-click .msp files and run as admin. Instead, I have to put the .msp file in a temp folder, open a command prompt elevated for that folder, and run the .msp file that way. Good grief!
• From a Microsoft Support article for an update for Windows 10: “This update includes reliability improvements to Windows update service components in Windows 10, versions 1507, 1511, 1607, 1703, and 1709… Only certain builds of Windows 10, versions 1507, 1511, 1607, and 1703 require this update.” Only certain builds? Which builds? What kind of criteria? Vague communications like this with regard to patches they release are useless to admins like us who manage networks with many different hardware configurations.

  

• From a Microsoft Support article for an update for Windows 10: “This update includes reliability improvements to Windows update service components in Windows 10, versions 1507, 1511, 1607, 1703, and 1709. It may also take steps to free up disk space on your device if you do not have enough disk space to install Windows updates.” You mean it’s going to delete some of my work files to make room for installing the update? Or just my temporary Internet files? Please be more specific when communicating with us what your patches are going to do to our machines!
• There’s something funny going on. When we try to patch our Windows Server 2016 systems, the smaller updates get applied but the larger cumulative updates fail. The ReportingEvents log suggests there’s some kind of timeout happening, but we can’t figure out why.
• We just installed the usual patches on the IIS servers in our test room and now we’re experiencing intermittent TLS connectivity issues when clients try and connect to these servers. When we uninstall the .NET patches the issues go away. This same thing happened last month as well. Fortunately, we discovered this on our test servers before rolling the patches out onto our production web servers!
• Just installed the latest cumulative update on several dozen Windows 10 Pro systems and discovered that a handful of them have lost about 50 GB disk space as a result. Looked further and discovered that the pagefile on these systems is huge. Did the CU do that? Why?
• I installed the v.1803 Feature Update for Windows 10 on my Surface Pro and instead of a logon screen got a black screen with the white mouse cursor. Nothing more happened and I had to force the machine to reboot by hitting the power button. Finally got the logon screen and logged on, but then instead of the desktop the screen went black again. Logged on again and this time it worked and I got the desktop. A few logons later though and it black screened again. A few weeks later I installed the latest Cumulative Update and now my machine seems to be behaving. Didn’t Microsoft test this Feature Update on their own hardware (Surface) before releasing it??
• Until now the KB4345419 patch was detected by WSUS as not applicable for Windows 10 v.1709 but suddenly it’s now been approved for deployment. Clearly, there must be something wrong with the detection logic at Microsoft’s end, maybe in the metadata for the patch.

Too many patches?

My thinking is that Microsoft may not have realized that the rapid release cadence they introduced with Windows 10 would add more complexity to their patch development process because instead of just patching one version of Windows they would have to test their patches against multiple versions like v.1703, 1709, and 1803. I also believe their strategy of releasing three kinds of patches for each software update — full, delta, and express updates — may also have contributed to both the confusion of their customers and the testing burden on their patch development team. In any case, the important thing for administrators is to understand how to best to handle the stresses associated with Patch Tuesday and I’ll deal with that subject soon in an upcoming article on this site.

Featured image: Shutterstock