Andrew Mercer
on this page

Patch and Upgrade Management

Patching is mostly not a technical problem. The commands are easy. What fails is scheduling, ordering, communication, and the ability to back out — so this guide focuses on those, across Windows Server and Linux, on Azure and off it.

For tool-specific automation on each OS, see the unattended update guides in this section:

For the Azure Update Manager platform (maintenance configurations, dynamic scopes, pre/post events), see Hybrid Server Operations.


Contents

  1. Patch policy
  2. Cadence and rings
  3. Designing maintenance windows
  4. Change management
  5. Pre-patch checks
  6. Reboot orchestration
  7. Rollback plans
  8. Post-patch validation
  9. Stakeholder communication
  10. Out-of-band and emergency patches
  11. End-of-life upgrades
  12. Third-party and application patching
  13. Metrics
  14. Monthly cycle at a glance

Patch policy

Write it down, get it approved, and point people to it. A one-page policy should state:

Item Example
Scope All Windows Server and Linux servers, Azure and non-Azure, all environments
Content Security and critical updates always; other updates quarterly or as needed; feature/major upgrades via project
Cadence Monthly cycle anchored to Patch Tuesday; Linux security updates within the same cycle
Compliance target 95% of servers fully patched (security/critical) within 30 days of release
Emergency KEV / actively exploited: per remediation SLAs
Windows Standard maintenance windows by ring; pre-approved standard change
Reboots Required updates must be followed by reboot within the window; no "pending reboot" > 7 days
Exclusions Only via approved exception
Rollback Every tier-0/1 server has a documented rollback method tested at least annually
Roles Platform team executes; app owners validate; change management approves

Cadence and rings

Monthly anchor

Microsoft releases security updates on the second Tuesday of each month ("Patch Tuesday"). Linux distros release continuously, but aligning Linux to the same monthly cycle keeps one rhythm for stakeholders — with emergency handling for anything urgent between cycles.

Rings

Ring Membership Timing after Patch Tuesday Purpose
Ring 0 – Lab Platform team's own test servers, golden-image build Day 0–1 Catch install failures, obvious breakage
Ring 1 – Non-prod Dev, test, QA, UAT Day 1–3 Application compatibility; app teams test
Ring 2 – Prod canary One or two servers per production role, least critical instances Day 4–7 Real workload, limited blast radius
Ring 3 – Prod broad Rest of production Day 8–14 Bulk
Ring 4 – Prod critical / manual Clusters needing coordinated failover, tier-0, change-frozen systems Day 10–21 Hands-on, attended windows

Gates between rings:

  • Install success rate ≥ 98% (failures understood).
  • No P1/P2 incidents attributed to the patch set.
  • App owner sign-off for ring 1 (explicit or silence-after-48h, as agreed).
  • Known-issues lists checked (Microsoft release health, distro errata, vendor advisories).

Non-prod should look like prod

If non-production doesn't run the same OS versions, agents, and baseline configuration as production, ring 1 tests nothing. Keep environment parity a standing requirement.


Designing maintenance windows

Inputs:

  • Business calendar — month-end close, payroll, retail peaks, academic terms, year-end freezes.
  • Service dependencies — what must stay up while something else reboots.
  • Time zones — "02:00" is someone's business hours. Patch per region if users are global.
  • Install + reboot duration — measure it. Windows CUs on older servers with slow disks can take 45+ minutes including reboot.
  • Staffing — attended windows for ring 4 need people awake. Don't schedule them at 03:00 Sunday without an on-call plan and compensation.

Window template:

Field Example
Name PROD-WAVE2-NA-EAST
Recurrence Monthly, Patch Tuesday + 11 days (second Saturday-ish)
Start / duration 02:00–06:00 America/Toronto
Scope patch-group=prod-wave2 AND region=canadacentral|onprem-kw1
Content Security + Critical (+ UpdateRollup on Windows)
Reboot If required
Pre-event Snapshot tier-1 disks; drain LB; silence monitors
Post-event Smoke tests; unsilence; post report
Owner platform-oncall
Freeze calendar Excludes 15 Dec – 5 Jan

Rules:

  • Windows reserve time for reboot. Azure Update Manager won't start installs late in a window; size accordingly.
  • Stagger within a window. Don't reboot every node of a role at the same minute — use separate waves or max-concurrency settings.
  • Hard stop. If the window ends, stop and reschedule; don't stretch into business hours without approval.

Change management

Routine monthly patching should be a standard change (pre-approved, low risk, well-understood procedure) so it doesn't need a CAB meeting every month. Earn that by:

  • A documented procedure (runbook), see Network Changes and Runbooks.
  • A track record of successful cycles.
  • Defined rollback.
  • Automated post-checks.

Use normal changes for: OS major upgrades, first-time patching of a newly onboarded system, anything touching tier-0 identity (domain controllers) outside the routine, cluster topology changes.

Use emergency changes for out-of-band security patches — approved quickly by the emergency CAB, reviewed retrospectively.


Pre-patch checks

Automate these in a pre-maintenance hook; fail the server out of the run if a critical check fails.

Common

  • [ ] Recent successful backup (and, for tier-1, a snapshot/restore point created now)
  • [ ] Free disk space (Windows: ≥ 10 GB on C: for CUs; Linux: /boot ≥ 100 MB free, /var ≥ 2 GB)
  • [ ] No pending reboot from an earlier run (or reboot first)
  • [ ] Cluster/replica health green
  • [ ] Monitoring maintenance mode set
  • [ ] Not in a change freeze or business blackout

Windows

# Pending reboot?
$pending = @(
  Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending'
  Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired'
  (Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager' -Name PendingFileRenameOperations -ErrorAction SilentlyContinue) -ne $null
) -contains $true

# Free space
$free = (Get-PSDrive C).Free / 1GB

# Component store health (slow; run weekly, not every window)
DISM /Online /Cleanup-Image /ScanHealth

[pscustomobject]@{ Host=$env:COMPUTERNAME; PendingReboot=$pending; FreeGB=[math]::Round($free,1) }

Linux

#!/usr/bin/env bash
set -euo pipefail
boot_free=$(df --output=avail -m /boot 2>/dev/null | tail -1 || echo 9999)
var_free=$(df --output=avail -m /var | tail -1)
reboot_req=no
[[ -f /var/run/reboot-required ]] && reboot_req=yes                          # Debian/Ubuntu
command -v needs-restarting >/dev/null && ! needs-restarting -r >/dev/null && reboot_req=yes   # RHEL
failed_units=$(systemctl --failed --no-legend | wc -l)
echo "boot_free_mb=$boot_free var_free_mb=$var_free reboot_required=$reboot_req failed_units=$failed_units"
(( boot_free < 100 )) && { echo "FAIL: /boot low"; exit 1; }
(( var_free < 2048 )) && { echo "FAIL: /var low"; exit 1; }
exit 0

Record the list of failed systemd units / stopped auto-start services before patching, so you don't blame the patch for something that was already broken.


Reboot orchestration

Ordering dependent services

Reboot in reverse dependency order going down and forward order coming up, or — better — patch one tier at a time while the others stay up:

Wave A: Secondary DB replicas / passive cluster nodes
Wave B: Fail over DB; patch former primaries
Wave C: App servers, half at a time behind the load balancer
Wave D: Web/front-end, half at a time
Wave E: Utility servers (jump hosts, schedulers, file servers)
Last:   Domain controllers, one per site at a time, never all at once

Load-balanced stateless tiers

  1. Remove node from LB pool (or set to drain).
  2. Wait for connections to drain (respect session timeouts).
  3. Patch + reboot.
  4. Health check passes → return to pool.
  5. Next node. Never take more than N - required capacity nodes out.

Azure Load Balancer / Application Gateway don't have a native "drain this VM" API in every SKU; common approaches are failing the health probe deliberately (e.g. a maintenance flag file the probe endpoint checks) or removing the NIC from the backend pool in the pre-event.

Windows Failover Clusters

Use Cluster-Aware Updating (CAU) — it drains roles, patches, reboots and resumes node by node. Configure the self-updating mode or drive it from a remote coordinator. Details in Unattended Updates on Windows Server.

SQL Server Always On availability groups

Patch secondaries first, fail over (planned manual failover to a synchronous secondary), patch the former primary, optionally fail back. Never patch the primary while a synchronous secondary is unhealthy.

Linux HA (Pacemaker/Corosync)

pcs node standby node2      # move resources off
# patch + reboot node2
pcs node unstandby node2
pcs status                  # confirm resources healthy before the next node

For Galera: one node at a time, wait for wsrep_local_state_comment = Synced before the next.

Kubernetes nodes

Covered by node image/OS auto-upgrade on AKS and kured elsewhere; see Container and Application Remediation and the Linux unattended upgrades guide. Ensure workloads have PodDisruptionBudgets so drains don't take out every replica.


Rollback plans

Every window needs a written answer to "what do we do if this breaks?" — before the window starts.

Rollback options by platform

Platform Option Notes
Azure VM Managed disk snapshot / restore point collection taken in pre-event Fastest whole-machine rollback. Application-consistent restore points for Windows (VSS) and Linux (pre/post scripts). Deleting snapshots after ~7 days controls cost.
On-prem VM Hypervisor snapshot (VMware/Hyper-V/libvirt) Short-lived only — don't leave snapshots running for weeks.
Physical server Backup restore; boot environment (ZFS/Btrfs); previous kernel Slowest; test it.
Windows update wusa /uninstall /kb:NNNNNNN for standalone updates; for combined SSU+LCU packages use DISM /Online /Get-Packages then DISM /Online /Remove-Package /PackageName:<name> The servicing stack component can't be removed. Uninstall leaves you vulnerable again — pair with an exception and a fix plan.
Windows (Known Issue Rollback) Microsoft can roll back specific non-security changes via KIR (automatic for many; Group Policy for enterprise-managed) Check release health before uninstalling a whole CU.
RHEL/Rocky/Alma dnf history list, dnf history undo <id>; boot previous kernel (grubby --set-default) Undo works only if the old packages are still available in repos/cache. Keep installonly_limit=3 for kernels.
Debian/Ubuntu apt install pkg=<oldversion> then apt-mark hold pkg; previous kernel from GRUB menu Old versions must be available (snapshot repos like snapshot.debian.org, or your own mirror).
SLES / openSUSE (Btrfs) snapper rollback Excellent if root is on Btrfs with snapper enabled.
FreeBSD Boot environments (bectl) See the FreeBSD guide.
Containers Previous image digest; kubectl rollout undo; App Service slot swap back See Container and Application Remediation.

Rollback decision criteria

Decide in advance what triggers rollback, so nobody debates it at 04:30:

  • Service health check failing for > 15 minutes after reboot, with no quick fix identified.
  • Data integrity risk of any kind → stop and roll back immediately.
  • Application owner declares business-critical function broken.

Rollback plan template

Change ID:
Servers / scope:
Rollback method: [snapshot restore | package downgrade | KB uninstall | previous kernel | slot swap]
Rollback point created: [snapshot IDs / restore point names / package versions recorded]
Time to roll back (estimated): 
Decision maker: 
Triggers: 
Steps:
  1.
  2.
Verification after rollback:
Follow-up: exception request for the un-applied patch (ID: )

Post-patch validation

Automate as much as possible in the post-event:

Check How
Machine booted and agent reporting Arc heartbeat / Azure VM agent status / monitoring agent
Patch result AUM installation result per machine; Get-HotFix; dnf history info last
Pending reboot cleared Same checks as pre-patch
Services running Compare auto-start services / systemd units against the pre-patch snapshot
Application health HTTP synthetic checks, /healthz endpoints, DB connect test
Error log spike Event log errors in last 30 min; journalctl -p err --since "-30min"
Scanner refresh Trigger on-demand assessment so the compliance view updates the same day

Then a human: app owners confirm for ring 1 and ring 4.


Stakeholder communication

People forgive outages they were warned about. Use the same templates every month so they become familiar.

Who to tell

Audience What they need Channel
App owners Which of their servers, when, expected impact, how to report issues Email + calendar invite; team channel
Service desk Schedule, expected symptoms, escalation path Ticket/knowledge article
End users (if visible impact) When, how long, workaround Status page / email
Change management Change record with plan and rollback ITSM
Leadership Monthly compliance summary only Report

Monthly pre-notice (send ~5 business days before ring 3)

Subject: [Scheduled maintenance] Server patching — <Month YYYY> — Prod wave 2 — <date> 02:00–06:00 ET

What: Monthly security patching (Windows <Month> cumulative updates; Linux security updates)
When: <Day, date>, 02:00–06:00 America/Toronto
Scope: <N> production servers in patch group prod-wave2 (list attached / dashboard link)
Impact: Individual servers reboot once; load-balanced services stay available.
        Services without redundancy (listed below) will be unavailable for ~20 minutes:
        - <service> (<server>)
Validation: Ring 1 (non-prod) patched <date> — no issues reported.
           Prod canaries patched <date> — no issues.
Rollback: Snapshots taken pre-patch for tier-1 servers.
Contact during window: platform on-call, <channel/phone>
Change: CHG<number>

Start / completion / issue notices

[STARTED] Prod wave 2 patching has started (02:00). Next update at completion.

[COMPLETED] Prod wave 2 patching completed at 04:12.
  Patched: 138/140   Failed: 2 (srv-a, srv-b — investigating, no service impact)
  Reboots: 131   Health checks: all passing
  Dashboard: <link>

[ISSUE] Prod wave 2: <service> failed health checks after reboot of <server> at 03:05.
  Impact: <what users see>
  Action: rolling back <server> from snapshot (decision 03:20). ETA 03:45.
  Next update: 03:50.

Out-of-band and emergency patches

When Microsoft ships an out-of-band update, or a Linux package fixes an exploited vulnerability:

  1. Confirm applicability and exposure (see Vulnerability Remediation).
  2. Emergency change, shortened rings: lab (hours) → non-prod (same day) → internet-facing prod → everything else.
  3. Use AUM one-time update with the specific KB/package included rather than waiting for a schedule.
  4. Communicate with the emergency template; be explicit about the threat driving urgency.
  5. After: fold the update into the next standard cycle; review whether rings were too slow.

End-of-life upgrades

EOL software stops receiving security fixes, so every unpatched vulnerability becomes permanent. EOL work is a project, not a patch window.

Track EOL dates

endoflife.date publishes a free API:

curl -s https://endoflife.date/api/windows-server.json | jq -r '.[] | [.cycle, .eol, .extendedSupport] | @tsv'
curl -s https://endoflife.date/api/ubuntu.json        | jq -r '.[] | select(.lts) | [.cycle, .eol, .extendedSupport] | @tsv'
curl -s https://endoflife.date/api/rhel.json          | jq -r '.[] | [.cycle, .support, .eol] | @tsv'

Join that against your inventory (OS name + version from Resource Graph or CMDB) to produce a 24-month EOL horizon report. Do the same for runtimes and databases: .NET, Java, Node.js, Python, PostgreSQL, SQL Server, Kubernetes versions.

The one most likely to bite next

Windows Server 2016 extended support ends January 12, 2027. After that, security updates come only through paid Extended Security Updates (up to three more years, enrolled through volume licensing or Azure Arc). If you still have 2016 in the estate today, you're in the "decide this week" zone: upgrade, rebuild, decommission, or buy ESU as a bridge with an exception and an end date.

Strategy options

Option When Pros Cons
Rebuild on new OS, redeploy app App has automated deployment; servers are cattle Clean, no carried-over cruft, baseline from day one Needs deployment automation and data migration
In-place upgrade Pets with complex local config, limited time Fast, preserves config Carries over cruft; must be on a supported path; harder rollback
Replatform (PaaS, containers) App can move to App Service, AKS, managed DB Removes OS patching burden entirely App changes
Retire Nobody can explain what it does Best outcome Requires the "scream test" with care
ESU / extended life Can't move before EOL Buys time Costs money; still needs a plan and exception

In-place upgrade paths

Windows Server: Windows Server 2025 supports in-place upgrade from 2012 R2 and later (up to four versions back); 2022 from 2012 R2/2016/2019. Azure VMs support in-place upgrade using an upgrade media managed disk attached to the VM — follow Microsoft's Azure-specific procedure (and snapshot first). Check application vendor support and run setup.exe /compat scanonly style compatibility checks before committing.

RHEL family: leapp (RHEL 8 → 9, 9 → 10); ELevate for Alma/Rocky/Oracle cross-upgrades.

sudo dnf install -y leapp-upgrade
sudo leapp preupgrade         # read /var/log/leapp/leapp-report.txt; fix inhibitors
sudo leapp upgrade
sudo reboot

Ubuntu: do-release-upgrade (LTS → next LTS once the first point release is out).

sudo apt update && sudo apt full-upgrade -y && sudo reboot
sudo do-release-upgrade      # interactive; use -f DistUpgradeViewNonInteractive cautiously

Debian: update sources to the next codename, apt full-upgrade in two stages per the release notes. Read the release notes every time.

EOL upgrade plan skeleton

1. Inventory: list of affected servers, owners, apps, dependencies
2. Decide per server: rebuild | in-place | replatform | retire | ESU-bridge
3. Vendor support: confirm app support on target OS
4. Pilot: one non-prod server per app; test plan executed; issues logged
5. Schedule: production waves with app owners; maintenance windows booked
6. Communication: stakeholders notified (template as above, plus a project kickoff)
7. Execute: snapshot → upgrade/rebuild → validate → monitor 1 week
8. Decommission old servers (rebuild path) — remove from DNS, backups, monitoring, CMDB, Arc
9. Close: findings tied to EOL closed; exceptions resolved

Third-party and application patching

OS patching tools don't patch everything:

Software Patching path
Microsoft products via Microsoft Update (SQL Server CUs, .NET on Windows) Enable "updates for other Microsoft products"; SQL CUs usually handled as their own change
Third-party Windows apps (Java, browsers, 7-Zip, agents) WinGet, Configuration Manager, Intune (for supported scenarios), or vendor updaters
Linux third-party repos (Docker, Grafana, PostgreSQL PGDG, Elastic) Include their origins in your patch config deliberately, or manage via dedicated changes
Language runtimes and app dependencies Application pipeline — see Container and Application Remediation
Appliances / firmware / BIOS / BMC Vendor tools; quarterly hardware maintenance windows

Firmware gets forgotten. BMC/iDRAC/iLO vulnerabilities are high-impact on physical servers in third-party data centres; include them in the scanner scope and the patch calendar.


Metrics

Metric Target example
Patch compliance (security/critical within 30 days) ≥ 95%
Install success rate per cycle ≥ 98%
Mean time to patch (release → installed), by ring Ring 3 ≤ 14 days
Servers with pending reboot > 7 days 0
Servers not assessed in last 3 days 0
Rollbacks per cycle Track and review each
Servers on EOL OS 0 without exception
Incidents caused by patching Track; review in retro

Monthly cycle at a glance

Day (relative to Patch Tuesday) Activity
−5 Confirm windows, freeze calendar, send pre-notice for ring 1
0 Patch Tuesday: review release notes, known issues, KEV additions; decide OOB emergencies
0–1 Ring 0 lab
1–3 Ring 1 non-prod; app owners test
3 Pre-notice for prod rings
4–7 Ring 2 prod canaries; gate review
8–14 Ring 3 prod broad
10–21 Ring 4 attended / clusters
21–25 Stragglers, failed installs, exceptions for anything not patched
28 Compliance report; retro on issues; update runbooks

References

  • Microsoft Security Update Guide — msrc.microsoft.com/update-guide
  • Windows release health — learn.microsoft.com/windows/release-health
  • Azure Update Manager — Microsoft Learn
  • Red Hat Leapp documentation — access.redhat.com
  • Ubuntu release upgrades — ubuntu.com/server/docs
  • endoflife.date API
in this section
* unattended updates/