Patch and Upgrade Management¶
Patching is mostly not a technical problem. The commands are easy. What fails is scheduling, ordering, communication, and the ability to back out — so this guide focuses on those, across Windows Server and Linux, on Azure and off it.
For tool-specific automation on each OS, see the unattended update guides in this section:
- Unattended Updates on Windows Server
- Unattended Upgrades on Linux Servers
- Unattended Updates on FreeBSD
For the Azure Update Manager platform (maintenance configurations, dynamic scopes, pre/post events), see Hybrid Server Operations.
Contents¶
- Patch policy
- Cadence and rings
- Designing maintenance windows
- Change management
- Pre-patch checks
- Reboot orchestration
- Rollback plans
- Post-patch validation
- Stakeholder communication
- Out-of-band and emergency patches
- End-of-life upgrades
- Third-party and application patching
- Metrics
- Monthly cycle at a glance
Patch policy¶
Write it down, get it approved, and point people to it. A one-page policy should state:
| Item | Example |
|---|---|
| Scope | All Windows Server and Linux servers, Azure and non-Azure, all environments |
| Content | Security and critical updates always; other updates quarterly or as needed; feature/major upgrades via project |
| Cadence | Monthly cycle anchored to Patch Tuesday; Linux security updates within the same cycle |
| Compliance target | 95% of servers fully patched (security/critical) within 30 days of release |
| Emergency | KEV / actively exploited: per remediation SLAs |
| Windows | Standard maintenance windows by ring; pre-approved standard change |
| Reboots | Required updates must be followed by reboot within the window; no "pending reboot" > 7 days |
| Exclusions | Only via approved exception |
| Rollback | Every tier-0/1 server has a documented rollback method tested at least annually |
| Roles | Platform team executes; app owners validate; change management approves |
Cadence and rings¶
Monthly anchor¶
Microsoft releases security updates on the second Tuesday of each month ("Patch Tuesday"). Linux distros release continuously, but aligning Linux to the same monthly cycle keeps one rhythm for stakeholders — with emergency handling for anything urgent between cycles.
Rings¶
| Ring | Membership | Timing after Patch Tuesday | Purpose |
|---|---|---|---|
| Ring 0 – Lab | Platform team's own test servers, golden-image build | Day 0–1 | Catch install failures, obvious breakage |
| Ring 1 – Non-prod | Dev, test, QA, UAT | Day 1–3 | Application compatibility; app teams test |
| Ring 2 – Prod canary | One or two servers per production role, least critical instances | Day 4–7 | Real workload, limited blast radius |
| Ring 3 – Prod broad | Rest of production | Day 8–14 | Bulk |
| Ring 4 – Prod critical / manual | Clusters needing coordinated failover, tier-0, change-frozen systems | Day 10–21 | Hands-on, attended windows |
Gates between rings:
- Install success rate ≥ 98% (failures understood).
- No P1/P2 incidents attributed to the patch set.
- App owner sign-off for ring 1 (explicit or silence-after-48h, as agreed).
- Known-issues lists checked (Microsoft release health, distro errata, vendor advisories).
Non-prod should look like prod¶
If non-production doesn't run the same OS versions, agents, and baseline configuration as production, ring 1 tests nothing. Keep environment parity a standing requirement.
Designing maintenance windows¶
Inputs:
- Business calendar — month-end close, payroll, retail peaks, academic terms, year-end freezes.
- Service dependencies — what must stay up while something else reboots.
- Time zones — "02:00" is someone's business hours. Patch per region if users are global.
- Install + reboot duration — measure it. Windows CUs on older servers with slow disks can take 45+ minutes including reboot.
- Staffing — attended windows for ring 4 need people awake. Don't schedule them at 03:00 Sunday without an on-call plan and compensation.
Window template:
| Field | Example |
|---|---|
| Name | PROD-WAVE2-NA-EAST |
| Recurrence | Monthly, Patch Tuesday + 11 days (second Saturday-ish) |
| Start / duration | 02:00–06:00 America/Toronto |
| Scope | patch-group=prod-wave2 AND region=canadacentral|onprem-kw1 |
| Content | Security + Critical (+ UpdateRollup on Windows) |
| Reboot | If required |
| Pre-event | Snapshot tier-1 disks; drain LB; silence monitors |
| Post-event | Smoke tests; unsilence; post report |
| Owner | platform-oncall |
| Freeze calendar | Excludes 15 Dec – 5 Jan |
Rules:
- Windows reserve time for reboot. Azure Update Manager won't start installs late in a window; size accordingly.
- Stagger within a window. Don't reboot every node of a role at the same minute — use separate waves or max-concurrency settings.
- Hard stop. If the window ends, stop and reschedule; don't stretch into business hours without approval.
Change management¶
Routine monthly patching should be a standard change (pre-approved, low risk, well-understood procedure) so it doesn't need a CAB meeting every month. Earn that by:
- A documented procedure (runbook), see Network Changes and Runbooks.
- A track record of successful cycles.
- Defined rollback.
- Automated post-checks.
Use normal changes for: OS major upgrades, first-time patching of a newly onboarded system, anything touching tier-0 identity (domain controllers) outside the routine, cluster topology changes.
Use emergency changes for out-of-band security patches — approved quickly by the emergency CAB, reviewed retrospectively.
Pre-patch checks¶
Automate these in a pre-maintenance hook; fail the server out of the run if a critical check fails.
Common¶
- [ ] Recent successful backup (and, for tier-1, a snapshot/restore point created now)
- [ ] Free disk space (Windows: ≥ 10 GB on C: for CUs; Linux:
/boot≥ 100 MB free,/var≥ 2 GB) - [ ] No pending reboot from an earlier run (or reboot first)
- [ ] Cluster/replica health green
- [ ] Monitoring maintenance mode set
- [ ] Not in a change freeze or business blackout
Windows¶
# Pending reboot?
$pending = @(
Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending'
Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired'
(Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager' -Name PendingFileRenameOperations -ErrorAction SilentlyContinue) -ne $null
) -contains $true
# Free space
$free = (Get-PSDrive C).Free / 1GB
# Component store health (slow; run weekly, not every window)
DISM /Online /Cleanup-Image /ScanHealth
[pscustomobject]@{ Host=$env:COMPUTERNAME; PendingReboot=$pending; FreeGB=[math]::Round($free,1) }
Linux¶
#!/usr/bin/env bash
set -euo pipefail
boot_free=$(df --output=avail -m /boot 2>/dev/null | tail -1 || echo 9999)
var_free=$(df --output=avail -m /var | tail -1)
reboot_req=no
[[ -f /var/run/reboot-required ]] && reboot_req=yes # Debian/Ubuntu
command -v needs-restarting >/dev/null && ! needs-restarting -r >/dev/null && reboot_req=yes # RHEL
failed_units=$(systemctl --failed --no-legend | wc -l)
echo "boot_free_mb=$boot_free var_free_mb=$var_free reboot_required=$reboot_req failed_units=$failed_units"
(( boot_free < 100 )) && { echo "FAIL: /boot low"; exit 1; }
(( var_free < 2048 )) && { echo "FAIL: /var low"; exit 1; }
exit 0
Record the list of failed systemd units / stopped auto-start services before patching, so you don't blame the patch for something that was already broken.
Reboot orchestration¶
Ordering dependent services¶
Reboot in reverse dependency order going down and forward order coming up, or — better — patch one tier at a time while the others stay up:
Wave A: Secondary DB replicas / passive cluster nodes
Wave B: Fail over DB; patch former primaries
Wave C: App servers, half at a time behind the load balancer
Wave D: Web/front-end, half at a time
Wave E: Utility servers (jump hosts, schedulers, file servers)
Last: Domain controllers, one per site at a time, never all at once
Load-balanced stateless tiers¶
- Remove node from LB pool (or set to drain).
- Wait for connections to drain (respect session timeouts).
- Patch + reboot.
- Health check passes → return to pool.
- Next node. Never take more than
N - required capacitynodes out.
Azure Load Balancer / Application Gateway don't have a native "drain this VM" API in every SKU; common approaches are failing the health probe deliberately (e.g. a maintenance flag file the probe endpoint checks) or removing the NIC from the backend pool in the pre-event.
Windows Failover Clusters¶
Use Cluster-Aware Updating (CAU) — it drains roles, patches, reboots and resumes node by node. Configure the self-updating mode or drive it from a remote coordinator. Details in Unattended Updates on Windows Server.
SQL Server Always On availability groups¶
Patch secondaries first, fail over (planned manual failover to a synchronous secondary), patch the former primary, optionally fail back. Never patch the primary while a synchronous secondary is unhealthy.
Linux HA (Pacemaker/Corosync)¶
pcs node standby node2 # move resources off
# patch + reboot node2
pcs node unstandby node2
pcs status # confirm resources healthy before the next node
For Galera: one node at a time, wait for wsrep_local_state_comment = Synced before the next.
Kubernetes nodes¶
Covered by node image/OS auto-upgrade on AKS and kured elsewhere; see Container and Application Remediation and the Linux unattended upgrades guide. Ensure workloads have PodDisruptionBudgets so drains don't take out every replica.
Rollback plans¶
Every window needs a written answer to "what do we do if this breaks?" — before the window starts.
Rollback options by platform¶
| Platform | Option | Notes |
|---|---|---|
| Azure VM | Managed disk snapshot / restore point collection taken in pre-event | Fastest whole-machine rollback. Application-consistent restore points for Windows (VSS) and Linux (pre/post scripts). Deleting snapshots after ~7 days controls cost. |
| On-prem VM | Hypervisor snapshot (VMware/Hyper-V/libvirt) | Short-lived only — don't leave snapshots running for weeks. |
| Physical server | Backup restore; boot environment (ZFS/Btrfs); previous kernel | Slowest; test it. |
| Windows update | wusa /uninstall /kb:NNNNNNN for standalone updates; for combined SSU+LCU packages use DISM /Online /Get-Packages then DISM /Online /Remove-Package /PackageName:<name> |
The servicing stack component can't be removed. Uninstall leaves you vulnerable again — pair with an exception and a fix plan. |
| Windows (Known Issue Rollback) | Microsoft can roll back specific non-security changes via KIR (automatic for many; Group Policy for enterprise-managed) | Check release health before uninstalling a whole CU. |
| RHEL/Rocky/Alma | dnf history list, dnf history undo <id>; boot previous kernel (grubby --set-default) |
Undo works only if the old packages are still available in repos/cache. Keep installonly_limit=3 for kernels. |
| Debian/Ubuntu | apt install pkg=<oldversion> then apt-mark hold pkg; previous kernel from GRUB menu |
Old versions must be available (snapshot repos like snapshot.debian.org, or your own mirror). |
| SLES / openSUSE (Btrfs) | snapper rollback |
Excellent if root is on Btrfs with snapper enabled. |
| FreeBSD | Boot environments (bectl) |
See the FreeBSD guide. |
| Containers | Previous image digest; kubectl rollout undo; App Service slot swap back |
See Container and Application Remediation. |
Rollback decision criteria¶
Decide in advance what triggers rollback, so nobody debates it at 04:30:
- Service health check failing for > 15 minutes after reboot, with no quick fix identified.
- Data integrity risk of any kind → stop and roll back immediately.
- Application owner declares business-critical function broken.
Rollback plan template¶
Change ID:
Servers / scope:
Rollback method: [snapshot restore | package downgrade | KB uninstall | previous kernel | slot swap]
Rollback point created: [snapshot IDs / restore point names / package versions recorded]
Time to roll back (estimated):
Decision maker:
Triggers:
Steps:
1.
2.
Verification after rollback:
Follow-up: exception request for the un-applied patch (ID: )
Post-patch validation¶
Automate as much as possible in the post-event:
| Check | How |
|---|---|
| Machine booted and agent reporting | Arc heartbeat / Azure VM agent status / monitoring agent |
| Patch result | AUM installation result per machine; Get-HotFix; dnf history info last |
| Pending reboot cleared | Same checks as pre-patch |
| Services running | Compare auto-start services / systemd units against the pre-patch snapshot |
| Application health | HTTP synthetic checks, /healthz endpoints, DB connect test |
| Error log spike | Event log errors in last 30 min; journalctl -p err --since "-30min" |
| Scanner refresh | Trigger on-demand assessment so the compliance view updates the same day |
Then a human: app owners confirm for ring 1 and ring 4.
Stakeholder communication¶
People forgive outages they were warned about. Use the same templates every month so they become familiar.
Who to tell¶
| Audience | What they need | Channel |
|---|---|---|
| App owners | Which of their servers, when, expected impact, how to report issues | Email + calendar invite; team channel |
| Service desk | Schedule, expected symptoms, escalation path | Ticket/knowledge article |
| End users (if visible impact) | When, how long, workaround | Status page / email |
| Change management | Change record with plan and rollback | ITSM |
| Leadership | Monthly compliance summary only | Report |
Monthly pre-notice (send ~5 business days before ring 3)¶
Subject: [Scheduled maintenance] Server patching — <Month YYYY> — Prod wave 2 — <date> 02:00–06:00 ET
What: Monthly security patching (Windows <Month> cumulative updates; Linux security updates)
When: <Day, date>, 02:00–06:00 America/Toronto
Scope: <N> production servers in patch group prod-wave2 (list attached / dashboard link)
Impact: Individual servers reboot once; load-balanced services stay available.
Services without redundancy (listed below) will be unavailable for ~20 minutes:
- <service> (<server>)
Validation: Ring 1 (non-prod) patched <date> — no issues reported.
Prod canaries patched <date> — no issues.
Rollback: Snapshots taken pre-patch for tier-1 servers.
Contact during window: platform on-call, <channel/phone>
Change: CHG<number>
Start / completion / issue notices¶
[STARTED] Prod wave 2 patching has started (02:00). Next update at completion.
[COMPLETED] Prod wave 2 patching completed at 04:12.
Patched: 138/140 Failed: 2 (srv-a, srv-b — investigating, no service impact)
Reboots: 131 Health checks: all passing
Dashboard: <link>
[ISSUE] Prod wave 2: <service> failed health checks after reboot of <server> at 03:05.
Impact: <what users see>
Action: rolling back <server> from snapshot (decision 03:20). ETA 03:45.
Next update: 03:50.
Out-of-band and emergency patches¶
When Microsoft ships an out-of-band update, or a Linux package fixes an exploited vulnerability:
- Confirm applicability and exposure (see Vulnerability Remediation).
- Emergency change, shortened rings: lab (hours) → non-prod (same day) → internet-facing prod → everything else.
- Use AUM one-time update with the specific KB/package included rather than waiting for a schedule.
- Communicate with the emergency template; be explicit about the threat driving urgency.
- After: fold the update into the next standard cycle; review whether rings were too slow.
End-of-life upgrades¶
EOL software stops receiving security fixes, so every unpatched vulnerability becomes permanent. EOL work is a project, not a patch window.
Track EOL dates¶
endoflife.date publishes a free API:
curl -s https://endoflife.date/api/windows-server.json | jq -r '.[] | [.cycle, .eol, .extendedSupport] | @tsv'
curl -s https://endoflife.date/api/ubuntu.json | jq -r '.[] | select(.lts) | [.cycle, .eol, .extendedSupport] | @tsv'
curl -s https://endoflife.date/api/rhel.json | jq -r '.[] | [.cycle, .support, .eol] | @tsv'
Join that against your inventory (OS name + version from Resource Graph or CMDB) to produce a 24-month EOL horizon report. Do the same for runtimes and databases: .NET, Java, Node.js, Python, PostgreSQL, SQL Server, Kubernetes versions.
The one most likely to bite next¶
Windows Server 2016 extended support ends January 12, 2027. After that, security updates come only through paid Extended Security Updates (up to three more years, enrolled through volume licensing or Azure Arc). If you still have 2016 in the estate today, you're in the "decide this week" zone: upgrade, rebuild, decommission, or buy ESU as a bridge with an exception and an end date.
Strategy options¶
| Option | When | Pros | Cons |
|---|---|---|---|
| Rebuild on new OS, redeploy app | App has automated deployment; servers are cattle | Clean, no carried-over cruft, baseline from day one | Needs deployment automation and data migration |
| In-place upgrade | Pets with complex local config, limited time | Fast, preserves config | Carries over cruft; must be on a supported path; harder rollback |
| Replatform (PaaS, containers) | App can move to App Service, AKS, managed DB | Removes OS patching burden entirely | App changes |
| Retire | Nobody can explain what it does | Best outcome | Requires the "scream test" with care |
| ESU / extended life | Can't move before EOL | Buys time | Costs money; still needs a plan and exception |
In-place upgrade paths¶
Windows Server: Windows Server 2025 supports in-place upgrade from 2012 R2 and later (up to four versions back); 2022 from 2012 R2/2016/2019. Azure VMs support in-place upgrade using an upgrade media managed disk attached to the VM — follow Microsoft's Azure-specific procedure (and snapshot first). Check application vendor support and run setup.exe /compat scanonly style compatibility checks before committing.
RHEL family: leapp (RHEL 8 → 9, 9 → 10); ELevate for Alma/Rocky/Oracle cross-upgrades.
sudo dnf install -y leapp-upgrade
sudo leapp preupgrade # read /var/log/leapp/leapp-report.txt; fix inhibitors
sudo leapp upgrade
sudo reboot
Ubuntu: do-release-upgrade (LTS → next LTS once the first point release is out).
sudo apt update && sudo apt full-upgrade -y && sudo reboot
sudo do-release-upgrade # interactive; use -f DistUpgradeViewNonInteractive cautiously
Debian: update sources to the next codename, apt full-upgrade in two stages per the release notes. Read the release notes every time.
EOL upgrade plan skeleton¶
1. Inventory: list of affected servers, owners, apps, dependencies
2. Decide per server: rebuild | in-place | replatform | retire | ESU-bridge
3. Vendor support: confirm app support on target OS
4. Pilot: one non-prod server per app; test plan executed; issues logged
5. Schedule: production waves with app owners; maintenance windows booked
6. Communication: stakeholders notified (template as above, plus a project kickoff)
7. Execute: snapshot → upgrade/rebuild → validate → monitor 1 week
8. Decommission old servers (rebuild path) — remove from DNS, backups, monitoring, CMDB, Arc
9. Close: findings tied to EOL closed; exceptions resolved
Third-party and application patching¶
OS patching tools don't patch everything:
| Software | Patching path |
|---|---|
| Microsoft products via Microsoft Update (SQL Server CUs, .NET on Windows) | Enable "updates for other Microsoft products"; SQL CUs usually handled as their own change |
| Third-party Windows apps (Java, browsers, 7-Zip, agents) | WinGet, Configuration Manager, Intune (for supported scenarios), or vendor updaters |
| Linux third-party repos (Docker, Grafana, PostgreSQL PGDG, Elastic) | Include their origins in your patch config deliberately, or manage via dedicated changes |
| Language runtimes and app dependencies | Application pipeline — see Container and Application Remediation |
| Appliances / firmware / BIOS / BMC | Vendor tools; quarterly hardware maintenance windows |
Firmware gets forgotten. BMC/iDRAC/iLO vulnerabilities are high-impact on physical servers in third-party data centres; include them in the scanner scope and the patch calendar.
Metrics¶
| Metric | Target example |
|---|---|
| Patch compliance (security/critical within 30 days) | ≥ 95% |
| Install success rate per cycle | ≥ 98% |
| Mean time to patch (release → installed), by ring | Ring 3 ≤ 14 days |
| Servers with pending reboot > 7 days | 0 |
| Servers not assessed in last 3 days | 0 |
| Rollbacks per cycle | Track and review each |
| Servers on EOL OS | 0 without exception |
| Incidents caused by patching | Track; review in retro |
Monthly cycle at a glance¶
| Day (relative to Patch Tuesday) | Activity |
|---|---|
| −5 | Confirm windows, freeze calendar, send pre-notice for ring 1 |
| 0 | Patch Tuesday: review release notes, known issues, KEV additions; decide OOB emergencies |
| 0–1 | Ring 0 lab |
| 1–3 | Ring 1 non-prod; app owners test |
| 3 | Pre-notice for prod rings |
| 4–7 | Ring 2 prod canaries; gate review |
| 8–14 | Ring 3 prod broad |
| 10–21 | Ring 4 attended / clusters |
| 21–25 | Stragglers, failed installs, exceptions for anything not patched |
| 28 | Compliance report; retro on issues; update runbooks |
References¶
- Microsoft Security Update Guide — msrc.microsoft.com/update-guide
- Windows release health — learn.microsoft.com/windows/release-health
- Azure Update Manager — Microsoft Learn
- Red Hat Leapp documentation — access.redhat.com
- Ubuntu release upgrades — ubuntu.com/server/docs
- endoflife.date API