Achieving world-class Windows monthly patching efficiency

You're responsible for patch management in your organization. Do you have the right approach to achieve world-class protection of the devices under your care?

The key to keeping devices in your environment protected from today’s threats is to patch them as quickly as possible. On the second Tuesday of every month—also known as Update Tuesday, Patch Tuesday, or the “B” release—we release security updates for all Windows endpoints, including desktops, laptops, and servers. These updates provide consumers and IT administrators alike with the necessary security fixes to protect their devices before unpatched vulnerabilities can potentially be exploited.

Over the last few years, we have seen IT professionals and chief information security officers (CISOs) place greater attention on accelerating device updates across their organizations and implementing formal policies for updates and patches. Ever wonder how the most effective organizations have transformed their update management processes to get world-class results with the least impact on their users? This article describes our experience engaging deeply with customers on this topic and seeing them dramatically improve their results by embracing a “cloud cadence” mindset and transforming their approach to monthly servicing. This approach becomes increasingly relevant as organizations respond and adapt to the challenges of remote work, and business continuity more generally.

Later in this article, I'm going to deconstruct what a “cloud cadence” mindset means, go deeper into the update process and examine the impact of key factors on monthly patching performance such as how devices receive updates and what happens during each phase of the update process. I’ll offer tips on how you can improve the update adoption rate for devices you manage. While I focus on Windows 10, since that is where most commercial devices are today, all applies equally to Windows 11.

Fundamentally, there are two key transformations you can make to increase the efficiency of update management in your organization:

1. Streamline your review and approval processes. This is what we call a “cloud cadence” mindset and it is fundamental to how fast your devices can “get off the starting block” in their race to get patched before bad actors exploit vulnerabilities.
2. Tune the efficiency of the update machinery. Windows Update provides many policies to exert administrator control over the system for very specific situations. Unfortunately, it is not always clear that a consequence of implementing certain policies in combination can be reduced patch velocity, prevention of patching all together, or a worse end user experience.

Let’s walk through why monthly patching matters, and how to implement these transformations in more detail.

Why effective monthly patching matters

Over time, we have seen an increase in common vulnerabilities and exposures (CVEs) month over month, notwithstanding seasonal variances. Each vulnerability represents a potential attack vector for bad actors looking to take advantage of unpatched environments. As security techniques continue to get more sophisticated and bad actors are releasing exploits quicker, IT administrators must become more diligent about rapidly deploying servicing updates.

Common vulnerabilities and exposures: July 2019 – July 2021 (Source: Microsoft Security Response Center)

Also consider that while it is good that Microsoft releases newly discovered security CVE fixes each month, doing so makes the nature of each of the original vulnerabilities discoverable to bad actors if they were not previously aware of them. It is a common nefarious practice to reverse engineer fixes to learn how to exploit devices that are slow to patch. If a CVE is particularly severe, we have seen live exploits released within 48 hours by bad actors, so it is important to patch your organization’s devices quickly!

Adopting cloud-based monthly patching strategies

If your organization is subject to regulatory requirements for security patching, or your security organization has established internal requirements, you may already be working to reduce the time required to reach a specified monthly update adoption target or threshold.

Organizations that transform their update process as described later can achieve world-class update adoption each month using endpoint management tools of choice. Microsoft Endpoint Manager builds atop Windows manageability infrastructure to simplify the mechanics of this transformation, whether using Microsoft Endpoint Configuration Manager, or Microsoft Intune. If your organization cannot devote expertise to update management or if you just want to simplify, I recommend you take a fresh look at a cloud-managed distribution model for monthly updates using Microsoft Intune or co-management. The balance of this article focuses on cloud-managed distribution of updates.

To serve as an example, we’ll define three segments and compare monthly update adoption results:  

• Microsoft Update managed. These are devices connected to the Windows Update service but have no configured administrative policies which govern updates. Thus, updates are managed by the Windows Update service. A majority are consumer devices, many of which are used on an irregular schedule, often offline, and hard to update.
• Commercial self-managed, with no update process transformation. These devices are actively managed by IT administrators using on-premise infrastructure that is typically based on Windows Server Update Services (WSUS), but update policies are not configured with Microsoft’s best practices for a “cloud cadence” mindset.
• Commercial cloud-managed or self-managed with update process transformation. These devices are actively managed by IT administrators using Microsoft Endpoint Configuration Manager or Microsoft Intune (backed by Windows Update for Business), where update policies are configured with Microsoft’s best practices for a “cloud cadence” mindset.

Next, we’ll examine the average update adoption rate among these three segments from December 2020 through February 2021 for the many tens-of-millions of devices that are sending telemetry.

Device segment	T+14 days	T+28 days
Microsoft Update managed	73.2%	78.8%
Commercial self-managed, with no update process transformation	55.0%	80.0%
Commercial cloud-managed or self-managed with update process transformation	92.5%	93.6%

Compared to commercial self-managed devices where update process transformation has not occurred, we observe a significantly faster update adoption rate at T+14 days among devices managed by the Windows Update service and a far better rate for organizations who have adopted update process transformation, made simple by Microsoft Intune and Windows Update for Business. Both these configurations take advantage of best practices for update management.

Now, you may be curious: how are organizations achieving greater than 90% update success for commercial cloud-managed devices? How can you do the same? First, you need to fundamentally commit to the “cloud cadence” mindset. Second, you need to eliminate often-misunderstood policies that appear to tune update behavior but really result in slowing down the update machinery, often to the detriment of both velocity and the user experience. To support your effort to increase update velocity in your own organization, we have published the Windows 10 Update Baseline, which provides recommended configurations and tuning guidance designed to maximize device updates in the shortest time possible.

Streamline your review and approval processes

The number of vulnerabilities is increasing across all software, not just the operating system. To keep ahead of these risks, you must accelerate how quickly you protect your devices. You need to adopt strategies and processes that allow for more rapid deployment of monthly servicing patches safely. 

The first, most vital thing you should do is to reimagine how you review and approve monthly servicing updates. We call this adopting a “cloud cadence” mindset. This is less about technology and more about making a cultural shift away from legacy deployment approaches to service management maturity. So what does a cloud cadence mindset really mean?

Let's look at an analogy in the form of a technology introduced more than a hundred years ago: the automotive assembly line. When Henry Ford introduced the first automotive assembly line in 1913, it took more than twelve hours to assemble a Ford Model T automobile. With the introduction of the assembly line, that time was reduced to a mere two and a half hours, a six-fold improvement! None of the steps to assemble the car really changed, just the mindset and the approach for how to get there faster with the same personnel (although there were certainly further process optimizations that leveraged the assembly line to an even greater degree once the new approach was understood).

The same transformation can be achieved when managing updates in your organization. You can treat each update—a car in this analogy—as a project that must be fully reviewed, approved, and deployed as a single project before moving on to the next project. Or, for greater efficiency, you can adopt an approach where updates are constantly flowing. You implement stations along the assembly line for specific tasks, but the update consistently moves towards full and broad deployment. If you detect an issue, hit the “pause” button on your assembly line to diagnose and remediate the issue. Once remediated and you have high confidence that the issue has been addressed, you then resume the deployment process.

We call these different stations along the update deployment process rings. Dividing devices in your organization into groups, you order them from highest to lowest tolerance for issues. You steadily progress through these rings, carefully watching signals received (like the assembly line automation manager) should a pause be necessary. What we have heard from organizations that have embraced this approach is that it saves their organization a lot of money. We've also heard that this approach has a lower impact on overall resources as they spend time only on real issues encountered during a responsible staged rollout that leverages update rings versus a heavy and lengthy QA validation phase prior to deployment with their previous processes. Key to the success of this approach is to build and maintain highly effective listening mechanisms to surface any anomalies quickly, minimizing the impact of those anomalies on the rest of the update ring.

To learn more about how to craft and structure your update rings, check out Strategic and tactical considerations for ring-based Windows 10 deployments. 

Tune the efficiency of the update machinery

Even after adopting a cloud cadence mindset and getting great “off the starting block” results, there is more to do to increase the flow of approved updates to devices. Most of today’s commercial self-managed devices simply are not seeing anything close to the update velocity performance that is being achieved by both consumer and commercial devices that are receiving monthly updates via the cloud with the optimizations I mentioned above. The good news is that this is easily fixed.

There are three key factors to consider for monthly device updates:  

• Is the device active and connected? Does it successfully contact its update control point to learn of the update, and is it connected long enough to download and process the update?
• Is the device properly configured so as not to impede its ability to learn about, download, and install the update?
• Has a deadline policy been set? After connectivity, this is the most important consideration.

When working with self-managed commercial organizations, we consistently find that these three factors contribute to slower update adoption performance rates.

To achieve the fastest results, we recommend leveraging the cloud via the Windows Update for Business service to download and update monthly patches to devices. You can do this while still using Microsoft Endpoint Configuration Manager as your control plane and managing feature updates locally, by concurrently enrolling your devices into Microsoft Intune through co-management.

In this way, if a device has an internet connection, it will be notified of updates and download them from a Windows Update distribution point using the best download size optimization available. Otherwise, devices most often will need a VPN connection and to stay connected long enough for the update process to complete. If a device is running Windows 10, version 1803 or earlier and is not configured for express updates, it must be connected long enough to download and install any file greater than 1GB. As remote work scenarios are now more often the norm, this becomes a common reason why devices do not get updated quicker.

The device update process has four phases:

• Scan. A device checks Windows Update or your Windows Server Update Service endpoint at intervals, evaluating whether the update is appropriate by checking configurations (e.g., Group Policy or MDM policy) that have been set by the administrator.
• Download. Once the device determines that an update is available, it downloads it.
• Install. Following the download, depending on the device’s Windows Update settings, the update is installed on the system in the background.
• Commit and restart. Once installed, the device usually (but not always) may require a restart to complete installation update and implement any fixes or improvements included therein. 

The first three happen in the background, invisible to the user and not interrupting productivity. At each stage, there are opportunities to increase velocity via policy and settings. You can find a detailed description of these settings, and how best to optimize them, in the "Optimizing Windows 10 update adoption" guide included with the Windows 10 Update Baseline. If you decide not to deploy these recommendations, we strongly recommend you review the how-to guide for managing Windows updates for common configuration pitfalls you might be encountering within your organization's devices. As noted earlier, while Windows 10 is described throughout, everything applies equally to Windows 11.

By utilizing cloud-based updates and implementing recommended optimizations, we see significantly better update deployment among update populations managed by Microsoft. This includes employee devices at Microsoft, where we transitioned from traditional on-premises update management to cloud-delivered updates and optimized update delivery based on the recommendations in the Windows 10 Update Baseline.

Conclusion

We urge customers with self-managed environments who are struggling to achieve their monthly servicing goals to follow the same course, moving monthly servicing update delivery to the cloud and optimizing devices to as much as triple the average number that are fully patched by day 14.

To learn more about how Microsoft transitioned from traditional update management to cloud based co-management, see Keeping Windows 10 devices up to date with Microsoft Intune and Windows Update for Business. We also encourage you to check out these great sessions from Ignite:

• A simple recipe to accelerate Windows 10 patch compliance 
• Windows 10 update monitoring and reporting
• Security in the Microsoft Cloud Adoption Framework for Azure

 

Posted at https://sl.advdat.com/3eJWqat