Have Microsoft patches reached a painful tipping point?

When it comes to patching the Windows Server platform, various versions of Microsoft Windows, Microsoft Office, and other Microsoft applications, it almost seems to me like Microsoft is broken. In the old days, which in reality were not very long ago at all, those of us who work in the field managing or supporting Microsoft products felt confident that Microsoft thoroughly tested hotfixes, rollups, security updates, and other software patches before releasing them to customers. Sure, there were occasional times when something unexpected happened and a patch that fixed something ended up breaking something else either in Microsoft’s product line or for some third-party software vendor. But by and large, it seemed that patches got tested, tested, and tested again before they were let out of the barn door into the wild. But these days many of us wonder whether Microsoft — in its eagerness to push its customers away from on-premises deployments into their ever-expanding stable of cloud services like Azure and Office 365 — may have simply lost interest in investing the time, energy, and budget needed to ensure their development teams produce software updates that don’t cause customers more headaches than they solve.

In short, it almost feels like patching (including the development, testing, communications, and release of patches) at Microsoft is broken, or is in danger of soon reaching that state. And Patch Tuesday, that second Tuesday of each month when Microsoft releases security updates to fix bugs they’ve identified in their software products, has become as dreaded for many admins as Friday the 13th was for the Knights Templar when Philip IV arrested hundreds of them on Oct. 13, 1307. Below I’ve shared some examples of the kinds of things I’ve been hearing in recent months from frustrating admins who work with Microsoft products and patch Windows-based infrastructures ranging from a handful of computers to tens of thousands of deployed systems. In a future article here on TechGenix, I’ll discuss some mitigation strategies you can follow and best practices you can adhere to reduce the potential risk of a nightmare happening when you patch your organization’s Windows clients and servers.

Stories from the field


Most of the below incidents have happened within the last three months, and it seems to many of us that patch development/management has been rapidly going downhill at Microsoft during this time period. To protect the identity of those involved I’ve fictionalized these incidents by altering personal details, but they’re all true stories in their technical essentials:

  • Installed the latest Security Monthly Quality Rollup for Windows 7 for x64-based Systems on my Dell Optiplex and it froze the system. This caused me much grief because it was my main work machine and it took me an hour to make it bootable again from a restore point. I’m sick of Microsoft making us their live testers for their software updates!!
  • Really? To address the Speculative Store Bypass (SSB) vulnerability in Windows with regard to the newly identified Spectre/Meltdown variants on Intel systems, I need to go through the steps of applying Windows updates that provide support for SSBD, then manually download and apply hardware/microcode updates from my device OEM, then manually change a registry setting to turn on Speculative Store Bypass Disable (SSBD) on the CPU, then run a Windows PowerShell script, then… C’mon! How can we be expected to do this manually for all the PCs we manage on our network?
  • From a Microsoft Support article for a rollup for Microsoft Exchange: “When you try to manually install this security update in ‘normal mode’ (not running the update as an administrator) and by double-clicking the update file (.msp), some files are not correctly updated. When this issue occurs, you do not receive an error message or any indication that the security update is not correctly installed… To avoid this issue, run the security update in elevated mode, as an administrator. To do this, right-click the update file, and then click Run as administrator.” Thanks again, Microsoft, because you can’t right-click .msp files and run as admin. Instead, I have to put the .msp file in a temp folder, open a command prompt elevated for that folder, and run the .msp file that way. Good grief!
  • From a Microsoft Support article for an update for Windows 10: “This update includes reliability improvements to Windows update service components in Windows 10, versions 1507, 1511, 1607, 1703, and 1709… Only certain builds of Windows 10, versions 1507, 1511, 1607, and 1703 require this update.” Only certain builds? Which builds? What kind of criteria? Vague communications like this with regard to patches they release are useless to admins like us who manage networks with many different hardware configurations.

    Pixabay

  • From a Microsoft Support article for an update for Windows 10: “This update includes reliability improvements to Windows update service components in Windows 10, versions 1507, 1511, 1607, 1703, and 1709. It may also take steps to free up disk space on your device if you do not have enough disk space to install Windows updates.” You mean it’s going to delete some of my work files to make room for installing the update? Or just my temporary Internet files? Please be more specific when communicating with us what your patches are going to do to our machines!
  • There’s something funny going on. When we try to patch our Windows Server 2016 systems, the smaller updates get applied but the larger cumulative updates fail. The ReportingEvents log suggests there’s some kind of timeout happening, but we can’t figure out why.
  • We just installed the usual patches on the IIS servers in our test room and now we’re experiencing intermittent TLS connectivity issues when clients try and connect to these servers. When we uninstall the .NET patches the issues go away. This same thing happened last month as well. Fortunately, we discovered this on our test servers before rolling the patches out onto our production web servers!
  • Just installed the latest cumulative update on several dozen Windows 10 Pro systems and discovered that a handful of them have lost about 50 GB disk space as a result. Looked further and discovered that the pagefile on these systems is huge. Did the CU do that? Why?
  • I installed the v.1803 Feature Update for Windows 10 on my Surface Pro and instead of a logon screen got a black screen with the white mouse cursor. Nothing more happened and I had to force the machine to reboot by hitting the power button. Finally got the logon screen and logged on, but then instead of the desktop the screen went black again. Logged on again and this time it worked and I got the desktop. A few logons later though and it black screened again. A few weeks later I installed the latest Cumulative Update and now my machine seems to be behaving. Didn’t Microsoft test this Feature Update on their own hardware (Surface) before releasing it??
  • Until now the KB4345419 patch was detected by WSUS as not applicable for Windows 10 v.1709 but suddenly it’s now been approved for deployment. Clearly, there must be something wrong with the detection logic at Microsoft’s end, maybe in the metadata for the patch.

Too many patches?

My thinking is that Microsoft may not have realized that the rapid release cadence they introduced with Windows 10 would add more complexity to their patch development process because instead of just patching one version of Windows they would have to test their patches against multiple versions like v.1703, 1709, and 1803. I also believe their strategy of releasing three kinds of patches for each software update — full, delta, and express updates — may also have contributed to both the confusion of their customers and the testing burden on their patch development team. In any case, the important thing for administrators is to understand how to best to handle the stresses associated with Patch Tuesday and I’ll deal with that subject soon in an upcoming article on this site.

Featured image: Shutterstock

Mitch Tulloch

Mitch Tulloch is Senior Editor of both WServerNews and FitITproNews and is a widely recognized expert on Windows Server and cloud technologies. He has written more than a thousand articles and has authored or been series editor for over 50 books for Microsoft Press and other publishers. Mitch has also been a twelve-time recipient of the Microsoft Most Valuable Professional (MVP) award in the technical category of Cloud and Datacenter Management. He currently runs an IT content development business in Winnipeg, Canada.

Share
Published by
Mitch Tulloch
Tags patch

Recent Posts

Hardware RAID vs. software RAID: Pros and cons for each

RAID is a technique to virtualize independent disks into arrays for improved performance. Should you…

3 days ago

After the plague: What IT will look like in a post-COVID-19 world

COVID-19 has changed everything, but once it disappears, we will not go back to how…

3 days ago

Solved: Outlook defaults to Microsoft 365 version with Exchange server

An Exchange server with a hybrid connection to Microsoft 365 is usually pretty seamless —…

4 days ago

How chatbots are changing the way teams communicate internally

Chatots are primarily thought of as consumer-facing solutions. They bring life to customer interactions by…

4 days ago

Hakbit ransomware campaign targeting specific European countries

The newly uncovered Hakbit ransomware campaign spread via spear-phishing emails may indicate a shift in…

4 days ago

Credential stuffing: Everything you need to know to avoid being a victim

Credential stuffing is yet another weapon being used by cybercriminals. Here’s what credential stuffing is…

5 days ago