Have Microsoft patches reached a painful tipping point?

When it comes to patching the Windows Server platform, various versions of Microsoft Windows, Microsoft Office, and other Microsoft applications, it almost seems to me like Microsoft is broken. In the old days, which in reality were not very long ago at all, those of us who work in the field managing or supporting Microsoft products felt confident that Microsoft thoroughly tested hotfixes, rollups, security updates, and other software patches before releasing them to customers. Sure, there were occasional times when something unexpected happened and a patch that fixed something ended up breaking something else either in Microsoft’s product line or for some third-party software vendor. But by and large, it seemed that patches got tested, tested, and tested again before they were let out of the barn door into the wild. But these days many of us wonder whether Microsoft — in its eagerness to push its customers away from on-premises deployments into their ever-expanding stable of cloud services like Azure and Office 365 — may have simply lost interest in investing the time, energy, and budget needed to ensure their development teams produce software updates that don’t cause customers more headaches than they solve.

In short, it almost feels like patching (including the development, testing, communications, and release of patches) at Microsoft is broken, or is in danger of soon reaching that state. And Patch Tuesday, that second Tuesday of each month when Microsoft releases security updates to fix bugs they’ve identified in their software products, has become as dreaded for many admins as Friday the 13th was for the Knights Templar when Philip IV arrested hundreds of them on Oct. 13, 1307. Below I’ve shared some examples of the kinds of things I’ve been hearing in recent months from frustrating admins who work with Microsoft products and patch Windows-based infrastructures ranging from a handful of computers to tens of thousands of deployed systems. In a future article here on TechGenix, I’ll discuss some mitigation strategies you can follow and best practices you can adhere to reduce the potential risk of a nightmare happening when you patch your organization’s Windows clients and servers.

Stories from the field

Most of the below incidents have happened within the last three months, and it seems to many of us that patch development/management has been rapidly going downhill at Microsoft during this time period. To protect the identity of those involved I’ve fictionalized these incidents by altering personal details, but they’re all true stories in their technical essentials:

  • Installed the latest Security Monthly Quality Rollup for Windows 7 for x64-based Systems on my Dell Optiplex and it froze the system. This caused me much grief because it was my main work machine and it took me an hour to make it bootable again from a restore point. I’m sick of Microsoft making us their live testers for their software updates!!
  • Really? To address the Speculative Store Bypass (SSB) vulnerability in Windows with regard to the newly identified Spectre/Meltdown variants on Intel systems, I need to go through the steps of applying Windows updates that provide support for SSBD, then manually download and apply hardware/microcode updates from my device OEM, then manually change a registry setting to turn on Speculative Store Bypass Disable (SSBD) on the CPU, then run a Windows PowerShell script, then… C’mon! How can we be expected to do this manually for all the PCs we manage on our network?
  • From a Microsoft Support article for a rollup for Microsoft Exchange: “When you try to manually install this security update in ‘normal mode’ (not running the update as an administrator) and by double-clicking the update file (.msp), some files are not correctly updated. When this issue occurs, you do not receive an error message or any indication that the security update is not correctly installed… To avoid this issue, run the security update in elevated mode, as an administrator. To do this, right-click the update file, and then click Run as administrator.” Thanks again, Microsoft, because you can’t right-click .msp files and run as admin. Instead, I have to put the .msp file in a temp folder, open a command prompt elevated for that folder, and run the .msp file that way. Good grief!
  • From a Microsoft Support article for an update for Windows 10: “This update includes reliability improvements to Windows update service components in Windows 10, versions 1507, 1511, 1607, 1703, and 1709… Only certain builds of Windows 10, versions 1507, 1511, 1607, and 1703 require this update.” Only certain builds? Which builds? What kind of criteria? Vague communications like this with regard to patches they release are useless to admins like us who manage networks with many different hardware configurations.

  • From a Microsoft Support article for an update for Windows 10: “This update includes reliability improvements to Windows update service components in Windows 10, versions 1507, 1511, 1607, 1703, and 1709. It may also take steps to free up disk space on your device if you do not have enough disk space to install Windows updates.” You mean it’s going to delete some of my work files to make room for installing the update? Or just my temporary Internet files? Please be more specific when communicating with us what your patches are going to do to our machines!
  • There’s something funny going on. When we try to patch our Windows Server 2016 systems, the smaller updates get applied but the larger cumulative updates fail. The ReportingEvents log suggests there’s some kind of timeout happening, but we can’t figure out why.
  • We just installed the usual patches on the IIS servers in our test room and now we’re experiencing intermittent TLS connectivity issues when clients try and connect to these servers. When we uninstall the .NET patches the issues go away. This same thing happened last month as well. Fortunately, we discovered this on our test servers before rolling the patches out onto our production web servers!
  • Just installed the latest cumulative update on several dozen Windows 10 Pro systems and discovered that a handful of them have lost about 50 GB disk space as a result. Looked further and discovered that the pagefile on these systems is huge. Did the CU do that? Why?
  • I installed the v.1803 Feature Update for Windows 10 on my Surface Pro and instead of a logon screen got a black screen with the white mouse cursor. Nothing more happened and I had to force the machine to reboot by hitting the power button. Finally got the logon screen and logged on, but then instead of the desktop the screen went black again. Logged on again and this time it worked and I got the desktop. A few logons later though and it black screened again. A few weeks later I installed the latest Cumulative Update and now my machine seems to be behaving. Didn’t Microsoft test this Feature Update on their own hardware (Surface) before releasing it??
  • Until now the KB4345419 patch was detected by WSUS as not applicable for Windows 10 v.1709 but suddenly it’s now been approved for deployment. Clearly, there must be something wrong with the detection logic at Microsoft’s end, maybe in the metadata for the patch.

Too many patches?

My thinking is that Microsoft may not have realized that the rapid release cadence they introduced with Windows 10 would add more complexity to their patch development process because instead of just patching one version of Windows they would have to test their patches against multiple versions like v.1703, 1709, and 1803. I also believe their strategy of releasing three kinds of patches for each software update — full, delta, and express updates — may also have contributed to both the confusion of their customers and the testing burden on their patch development team. In any case, the important thing for administrators is to understand how to best to handle the stresses associated with Patch Tuesday and I’ll deal with that subject soon in an upcoming article on this site.

Featured image: Shutterstock

About The Author

8 thoughts on “Have Microsoft patches reached a painful tipping point?”

  1. this has actually been happening for a fairly long time now, patches to Exchange server have been ‘ropey’ to say the least all the way back to Exchange 2013, and is something that is now becoming more apparent from the servers platforms to the ‘desktop’ level.

    I think it is directly related to the way Microsoft build and test their software , that is they do test builds, but they don’t actually bother to test the patching/upgrade process any more.

    In Office 365 (and Azure) Microsoft dont patch machines at all, they just ‘replace’ by re-imaging them, with the latest new builds when they are rolled-out.
    It makes it easier for them as provisioning new and imaging old (and replacing broken) servers in Azure is a simpler process.
    That is not realistic for small and medium size organisations, or individuals to replicate.
    They never actually test the patching process and (expect) let us do it for them, so we are left with the pain of patching over and over again…..

    1. I’m worried too about Microsoft’s increasingly poor QA and how it’s now affecting patches for their server products and not just client OS/apps. I wasn’t aware though of the problems with patching Exchange as I haven’t worked with that platform since Exchange 2010. –Mitch

  2. I don’t know anyone who ever trusted that MS updates won’t break anything. It’s kind of expected and as such I wait a while to apply updates.
    23 yrs experience of faulty MS product.

  3. Funny this article should arrive this week. Just last week MS shoved a Win10 update down my Dell workstation’s throat and upon reboot…it didn’t. Boot drive not available.

    Startup didn’t fix it. Neither did fixmbr. Or chkdsk. Thankfully I had made a ShadowProtect backup 10 days earlier. After four hours I was back in business.

    I used Group Policy to turn off the automatic updates and auto restart. And the next morning…same thing. This time I went straight to a restore and 20 minutes later I was back up. So I used GP again and made sure to reboot just in case that was necessary.

    And three hours later it did it again! Another restore (I was getting really good at this). I finally edited the registry to stop the madness, and that afternoon it updated again! This time, though, it came back up.

    This whole story stinks at many levels. Forced updates. Inability to stop the madness. Then an update that suddenly works?

    I gave up on Exchange years ago, again after a service pack trashed the setup. Rather than restore from backup I made the move to SmarterMail with the ActiveSync add-on. It’s not perfect, but updates are very regular and take five minutes to apply. That is one headache I no longer face.

    1. Matt thanks for your feedback and especially your comment about ShadowProtect, I’ll have to give that product a try sometime. –Mitch

  4. I have my system running windows 7 and ever since 9/12, i’ve been getting the capi2 error (event 4107) stating that the certificate is invalid. Now, I did
    run “certutil -urlcache * delete” command and the error still showed up. So I downloaded the certificate and in going through it, it says that the cert is valid from 9/12/17 to 9/12/18. So the error is valid and microsoft’s cert is invalid. Now just to test, I disabled the automatic time sync and changed the date of the system to 8/18/18 and lo and behold, the error did not appear. At this point, why hasn’t microsoft reissued a valid cert. How do we tell them to fix it?

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top