Servicing the Windows servicing stack: Welcome to the nightmare

Bootstrapping the laces on your boots is hard enough. And no one wants to walk around with their laces untied as you’re likely to trip and crack your skull. Keeping the servicing stack updated on your Windows and Windows Server systems is something else, however. It’s vitally important for the security of your Windows-based infrastructure. Unfortunately, it’s also becoming more and more of a challenge — and some would even say a nightmare.

Let’s try to get a handle on this subject. Microsoft uses the term “servicing” to refer to the task of installing a role, feature, hotfix, or software update on a Windows or Windows Server operating system installation. Online servicing happens while the operating system is running; a good example of this is when you download and install a security update or hotfix. Servicing can also happen offline when Windows isn’t running, for example when mounting a Windows image using DISM and apply an update to your image. We’ll focus in this article though only on online servicing when it refers to applying Windows updates.

Servicing stack: Just what is it?

Shutterstock

The term “servicing stack” refers to a group of Windows components that make servicing possible on Windows systems. These components include the component store (the %WinDir%\winsxs directory), the package store (the %WinDir%\servicing\packages directory), various executables and other system code, and the registry (specifically the HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing registry key). That’s the way it’s been since Windows Vista and Windows Server 2008 and it’s basically how things stand today with Windows 10 and Windows Server 2019 apart from some improved functionality that’s been added to the DISM command in later versions of Windows. Anyway, if you’re interested you can read more about how the Windows servicing stack works in these two blog posts by Joseph Conway who used to be known as The Windows Servicing Guy. He’s since moved on to other areas at Microsoft but his blog has not been taken down (thank goodness).

What’s problematic from an administrator’s perspective, however, is that Microsoft sometimes releases updates to the servicing stack for different versions of Windows. A servicing stack update (SSU) is basically a hotfix you apply to your Windows or Windows Server system to enhance or fix some aspect of the servicing stack on that installation. I can’t recall exactly when Microsoft started releasing out-of-band updates for their servicing stacks — I think there were one or two posts SP1 SSUs released for Windows 7 SP1 — but I do recall one in particular that Microsoft released in early 2014 that backported a new Component Store Footprint Reduction feature introduced in Windows 8.1 to the earlier Windows 8 operating system. Backporting new functionality to previous versions of Windows sounds like a good idea, and it is, generally.

But what happens when SSUs start flying left and right for each released version of Windows including every supported version of Windows 10? Well, the answer should be that everything is OK if there is a simple way of identifying which SSU goes with which Windows version and a reliable method of ensuring that the appropriate SSUs get installed on your Windows systems.

Why must there always be a problem?

Unfortunately, that’s where the troubles often begin. What Microsoft hasn’t seemed to consider recently is how updating the servicing stack can interfere with keeping your Windows client and server systems patched with the latest updates. The problem is that if a new SSU is released concurrently with the latest cumulative update (CU) for Windows, the SSU needs to be applied first before installing the CU. Unfortunately, until recently Microsoft has often been packaging SSUs together with CUs and if the SSU doesn’t install first for some reason, the CU install fails. And unfortunately the CU installs fails without notification of why it has fails; you don’t get a Windows Update notification saying “One or more updates failed to install because of an outdated servicing stack” or something similar. Instead, administrators have had to look up the Knowledge Base (KB) articles associated with the failed CU to read “Important When installing the servicing stack update (SSU) and the latest cumulative update (LCU), install the SSU first to mitigate potential issues while installing the LCU.” Well, thank you very much for not telling me that ahead of time!

This annoyance usually hasn’t affected ordinary users, however, because they typically install software updates directly by downloading them from Windows Update (WU), and in such cases, the SSU is typically installed first behind the scene before other updates are installed. Note, however, if you’re using Windows Update directly to automatically keep client systems patched, installed SSUs don’t show up in the display of Update History in Windows Update. Instead, you need to use DISM to verify that the latest SSU has been applied to your installation.

If instead you are a business or organization that’s using Windows Server Update Services (WSUS) or System Center Configuration Manager (SCCM) for keeping Windows updated, things usually work the same way — most of the time. Unfortunately, WSUS/SCCM seems to fail with regard to this from time to time, and the difficulty in troubleshooting what went wrong stems from the fact that packaging SSUs into CUs as it has happened in the past makes it difficult to identify whether an update for the servicing stack is required or been released for that month’s CUs. A big part of the problem seems to have been that Microsoft had been flagging SSUs as “critical” updates but not as “security” updates, though they obviously are important as far as security goes since they ensure you’ll be able to install the latest CUs on your systems.

This is one of the reasons behind the many fiascos that occurred last fall regarding updates Microsoft released for their Windows platforms. To address this issue John Wilcox, a Windows-as-a-service Evangelist at Microsoft, issued a comment on the Microsoft Tech Community last October saying, “To ensure our customers do not encounter this specific situation again, going forward, if we release a new servicing stack update, it will be marked as ‘security,’ not just ‘critical,’ so that it is included by those customers who are installing only tagged security fixes.” Hopefully, this has fixed most of the problems going forward with regard to keeping the Windows servicing stack up to date so your Windows client and server systems can be secure.

Administrators also need to be aware that when you manually download and install SSUs and CUs from the Microsoft Update Catalog as some of you do, you need to know which SSU to install on which Windows version and ensure you install the SSU before you install the LCU for that version. And you should also take special care if you use a non-Microsoft solution for Windows patch management such as PDQ Deploy, BatchPatch, SolarWinds, and so on. Make sure by testing that the patch management platform you are using applies the necessary and appropriate SSU to your Windows version before it tries to install the LCU for that version.

So now everything should be smooth sailing. But is it?

On Oct. 9 of last year, Microsoft released an SSU (KB3177467) for Windows 7 SP1 and Windows Server 2008 R2 SP1 that caused some problems for some users. When they installed the update together with other updates and a restart was required, they found their machines stuck on "Stage 2 of 2" or "Stage 3 of 3". To finish installing the SSU and associated updates they were informed in the KB article to press Ctrl+Alt+Del to continue to log on, and they were advised in managed environments like WSUS/SCCM that they could avoid these kinds of issues by deploying the SSU as a stand-alone update. So some manual work was involved for administrators. Fortunately, installing an SSU usually doesn’t require a reboot, which means that updating the servicing stack typically doesn’t cause any special outage for either your users or the Windows servers your business depends on. But if the SU is associated with a CU and the CU requires an update after installation, then some interruption is necessary. Anyway, what this situation tells us is that we’re not totally out of the woods yet.

Not only that, but WU users are still often saddled with unnecessary reboots that interrupt and delay their workflow. The user’s machine downloads the latest CU along with a new SSU and then tries to install the CU first. This fails of course — the SSU needs to be installed first. So the WU cycle runs again and the SSU gets installed this time, but the machine is not yet secure because the CCU hasn’t been installed. So the machine reboots and WU runs again and this time the CU installs. What a pain. I’m not saying this happens every month with every user for every Windows version, but it still happens frequently enough that users and those who administer the users complain about this on various forums. It happened to me before Christmas trying to install KB4471332 (CU) and KB4470788 (SSU) on my laptop running Windows 10 v.1809.

Microsoft being helpful?!

It would probably help too if Microsoft had been consistently informing us as administrators when new SSUs were released for each supported version of Windows. Well surprise, surprise, Microsoft has now started doing this since last September — see Microsoft Security Response Center security advisory ADV990001 for an updated list of the latest servicing stack updates for each operating system. Administrators would do well to bookmark this page and use it each month together with DISM.exe to ensure the latest SSUs has been installed on their systems in order to ensure the latest CUs have also been installed so they can be sure they’re systems are all fully patched.

Remember, if you don’t install the latest SSU you probably won’t be able to install the latest CU either, which means your system may be vulnerable since it’s not fully patched. There are also possible other telltale signs (bad things) that can happen if you don’t install the required SSU before trying to install the latest CU. For example, back in January of last year, this caused BSODs for some Windows users, and a similar situation the following month caused some users to lose USB functionality on their systems.

Going forward to avoid such problems completely, Microsoft could perhaps try doing something innovative like tagging SSUs with a unique new tag like “Servicing Stack Update” instead of the current tagging of “Security Update.” That would make life a whole lot easier for those of us who use WSUS/SCCM to keep our Windows client and server systems patched and up to date. In other words, it would help Windows admins all around if Microsoft would make an effort to fix the whole process by which SSUs are identified and released instead of simply making a few concessions toward WSUS/SSCM administrators. Since Windows and Windows Server (and also WSUS and SCSM) are Microsoft products, it clearly should be Microsoft’s responsibility to simplify patch management for these platforms. It shouldn’t be our responsibility as admins to figure out ways to work around the gaps Microsoft has in their patching processes, for example, the common workaround of using separate software update groups in ConfigMgr for SSUs and CUs and deploying the SSU a day before the CU.

And by the way, if you want to prod Microsoft toward doing something concrete about this whole matter, please go and upvote this suggestion on UserVoice. Thanks!

Featured image: Shutterstock

Mitch Tulloch

Mitch Tulloch is Senior Editor of both WServerNews and FitITproNews and is a widely recognized expert on Windows Server and cloud technologies. He has written more than a thousand articles and has authored or been series editor for over 50 books for Microsoft Press and other publishers. Mitch has also been a twelve-time recipient of the Microsoft Most Valuable Professional (MVP) award in the technical category of Cloud and Datacenter Management. He currently runs an IT content development business in Winnipeg, Canada.

Share
Published by
Mitch Tulloch

Recent Posts

Hardware RAID vs. software RAID: Pros and cons for each

RAID is a technique to virtualize independent disks into arrays for improved performance. Should you…

3 days ago

After the plague: What IT will look like in a post-COVID-19 world

COVID-19 has changed everything, but once it disappears, we will not go back to how…

3 days ago

Solved: Outlook defaults to Microsoft 365 version with Exchange server

An Exchange server with a hybrid connection to Microsoft 365 is usually pretty seamless —…

3 days ago

How chatbots are changing the way teams communicate internally

Chatots are primarily thought of as consumer-facing solutions. They bring life to customer interactions by…

4 days ago

Hakbit ransomware campaign targeting specific European countries

The newly uncovered Hakbit ransomware campaign spread via spear-phishing emails may indicate a shift in…

4 days ago

Credential stuffing: Everything you need to know to avoid being a victim

Credential stuffing is yet another weapon being used by cybercriminals. Here’s what credential stuffing is…

4 days ago