Hybrid Server Operations¶
A "hybrid estate" here means one team responsible for servers that live in very different places:
| Location | Examples | Control plane you get for free | What you have to add |
|---|---|---|---|
| Azure IaaS | Azure VMs, VM Scale Sets | Azure Resource Manager, VM agent, Azure Update Manager (no charge for Azure VMs) | Patch orchestration settings, maintenance configurations, policy |
| On-premises | Hyper-V, VMware, KVM, bare metal | Nothing Azure-native | Azure Arc Connected Machine agent (azcmagent) |
| Third-party data centre / other cloud | Colo racks, hosting providers, AWS/GCP VMs | The provider's console (if any) | Azure Arc agent, outbound network path to Azure |
The goal is one inventory, one patch schedule model, one compliance view, regardless of where the server physically runs. Azure Arc projects non-Azure machines into Azure Resource Manager as Microsoft.HybridCompute/machines resources, which lets Azure Update Manager, Azure Policy machine configuration, Defender for Servers and Resource Graph treat them like Azure VMs.
This guide covers the platform: onboarding, scheduling, hardening. The process side lives in the sibling guides: Patch and Upgrade Management, Configuration Management and Vulnerability Remediation.
Contents¶
- Build the inventory first
- Onboarding non-Azure servers with Azure Arc
- Azure Update Manager concepts
- Patch orchestration modes for Azure VMs
- Maintenance configurations and dynamic scopes
- Pre- and post-maintenance events
- Reporting with Azure Resource Graph
- Cost and licensing notes
- Hardening: choosing a baseline
- Hardening Windows Server
- Hardening Linux
- Auditing baselines at scale with machine configuration
- Rolling hardening out without breaking things
- Servers that can't be Arc-enabled
- Operational checklist
Build the inventory first¶
Every other activity — patching, scanning, exception tracking — fails quietly if the asset list is wrong. A server that isn't in inventory is a server nobody patches.
Minimum fields per server:
| Field | Why it matters |
|---|---|
| Hostname / resource ID | Join key across scanner, CMDB, Azure |
| Location (Azure region, DC, provider) | Network paths, maintenance windows, data residency |
| OS and version | EOL tracking, which patch tooling applies |
Environment (dev/test/prod) |
Patch ring assignment |
| Patch group / wave | Which maintenance configuration it belongs to |
| Business owner and technical owner | Who approves windows, who fixes app breakage |
| Criticality (tier 0–3) | Remediation SLA and change process |
| Internet-exposed (yes/no) | Vulnerability prioritisation |
| Reboot constraints | Clustered? Stateful? Needs manual failover? |
Enforce the tag set with Azure Policy (Require a tag on resources / Inherit a tag from the resource group) so new Azure VMs and Arc machines can't land untagged. A suggested tag schema:
env = prod | test | dev
patch-group = prod-wave1 | prod-wave2 | nonprod
owner = [email protected]
criticality = tier0 | tier1 | tier2 | tier3
exposure = internet | internal
reboot-policy = auto | coordinated | manual
Reconcile three lists at least monthly: the CMDB, Azure Resource Graph (Azure VMs + Arc machines), and the vulnerability scanner's asset list. Anything that appears in only one list is either a ghost or an unmanaged server — both are findings.
Onboarding non-Azure servers with Azure Arc¶
Prerequisites¶
- Supported OS: Windows Server 2012 R2 and later; mainstream Linux distros (RHEL, Rocky, Alma, Oracle, Ubuntu, Debian, SLES, Amazon Linux, Azure Linux). Check the current support matrix before onboarding anything unusual.
- Outbound HTTPS (TCP 443) from the server to Azure endpoints — directly, through a proxy, or privately through an Azure Arc Private Link Scope over ExpressRoute/VPN.
- Resource providers registered in the target subscription:
az provider register --namespace Microsoft.HybridCompute
az provider register --namespace Microsoft.GuestConfiguration
az provider register --namespace Microsoft.HybridConnectivity
az provider register --namespace Microsoft.AzureArcData
- A service principal (or managed identity for automation) with the Azure Connected Machine Onboarding role on the target resource group — nothing broader.
Interactive onboarding (one server)¶
The portal (Azure Arc > Machines > Add/Create > Add a machine) generates a script. Under the hood it installs the agent and runs azcmagent connect.
At-scale onboarding¶
Linux:
# Install the agent (script from https://aka.ms/azcmagent; review before running)
curl -fsSL -o install_linux_azcmagent.sh https://gbl.his.arc.azure.com/azcmagent-linux
sudo bash install_linux_azcmagent.sh
sudo azcmagent connect \
--service-principal-id "$ARC_SP_ID" \
--service-principal-secret "$ARC_SP_SECRET" \
--tenant-id "$TENANT_ID" \
--subscription-id "$SUBSCRIPTION_ID" \
--resource-group "rg-arc-servers-dc1" \
--location "canadacentral" \
--cloud "AzureCloud" \
--tags "env=prod,patch-group=prod-wave2,[email protected],criticality=tier1"
Windows (elevated PowerShell):
# Download and install the agent MSI
Invoke-WebRequest -Uri "https://aka.ms/AzureConnectedMachineAgent" -OutFile "$env:TEMP\AzureConnectedMachineAgent.msi"
msiexec /i "$env:TEMP\AzureConnectedMachineAgent.msi" /qn /l*v "$env:TEMP\azcmagent-install.log"
& "$env:ProgramFiles\AzureConnectedMachineAgent\azcmagent.exe" connect `
--service-principal-id $env:ARC_SP_ID `
--service-principal-secret $env:ARC_SP_SECRET `
--tenant-id $env:TENANT_ID `
--subscription-id $env:SUBSCRIPTION_ID `
--resource-group "rg-arc-servers-dc1" `
--location "canadacentral" `
--tags "env=prod,patch-group=prod-wave2,[email protected],criticality=tier1"
Push the same script out with whatever you already have: Ansible, Group Policy startup script, Configuration Manager, or your hosting provider's automation. Don't put the SP secret in the script file — pass it in from Ansible Vault, a CI secret, or retrieve it from Key Vault at run time, and rotate it after the onboarding campaign.
Ansible example¶
- name: Onboard servers to Azure Arc
hosts: arc_onboarding
become: true
vars:
arc_rg: rg-arc-servers-dc1
arc_location: canadacentral
tasks:
- name: Check if already connected
ansible.builtin.command: azcmagent show --json
register: arc_show
changed_when: false
failed_when: false
- name: Download installer
ansible.builtin.get_url:
url: https://gbl.his.arc.azure.com/azcmagent-linux
dest: /tmp/install_linux_azcmagent.sh
mode: "0700"
when: arc_show.rc != 0
- name: Install agent
ansible.builtin.command: bash /tmp/install_linux_azcmagent.sh
when: arc_show.rc != 0
- name: Connect
ansible.builtin.command: >
azcmagent connect
--service-principal-id {{ arc_sp_id }}
--service-principal-secret {{ arc_sp_secret }}
--tenant-id {{ tenant_id }}
--subscription-id {{ subscription_id }}
--resource-group {{ arc_rg }}
--location {{ arc_location }}
--tags "env={{ env }},patch-group={{ patch_group }},owner={{ owner }}"
no_log: true
when: arc_show.rc != 0 or (arc_show.stdout | from_json).status != "Connected"
Verify and troubleshoot¶
azcmagent show # status, resource ID, last heartbeat
azcmagent check --location canadacentral # endpoint reachability test
azcmagent logs # bundle logs for support
azcmagent config list # proxy, extension allow/block lists
Lock the agent down: by default Arc allows remote extension installs. On sensitive servers, restrict which extensions may run:
azcmagent config set extensions.allowlist "Microsoft.CPlat.Core/WindowsPatchExtension,Microsoft.CPlat.Core/LinuxPatchExtension,Microsoft.GuestConfiguration/ConfigurationforLinux,Microsoft.Azure.AzureDefenderForServers/MDE.Linux"
azcmagent config set guestconfiguration.enabled true
Arc for VMware vSphere / SCVMM¶
If on-premises is mostly vSphere or SCVMM, Arc-enabled VMware vSphere and Arc-enabled SCVMM add VM lifecycle operations (start/stop/create) on top of the guest agent. Patching and policy still go through the guest-level Connected Machine agent described above.
Azure Update Manager concepts¶
Azure Update Manager (AUM) replaced Automation Update Management (retired August 2024) and needs no Log Analytics workspace or Automation account. It uses the WindowsPatchExtension / LinuxPatchExtension on each machine and drives the native package manager (Windows Update Agent, apt, dnf/yum, zypper).
| Concept | What it does |
|---|---|
| Assessment | Scans for missing updates. On-demand, or periodic assessment every 24 hours (enable via policy). |
| One-time update | Install now, ad hoc. Good for emergency patches. |
| Maintenance configuration | A Microsoft.Maintenance/maintenanceConfigurations resource: schedule + what to install + reboot behaviour. |
| Assignment | Links machines to a maintenance configuration — statically (by resource ID) or via a dynamic scope (by subscription, RG, location, OS, tags). |
| Pre/post events | Event Grid events fired before and after a schedule runs, so you can start VMs, drain load balancers, snapshot disks, run smoke tests. |
| Hotpatch | Rebootless security updates for supported Windows Server editions (Azure Edition on Azure; Windows Server 2025 via Arc). |
Turn on periodic assessment fleet-wide so the compliance view is never stale. The built-in policies are:
- Configure periodic checking for missing system updates on azure virtual machines
- Configure periodic checking for missing system updates on azure Arc-enabled servers
Assign both at management group scope with a DeployIfNotExists/Modify effect and a remediation task for existing machines.
Patch orchestration modes for Azure VMs¶
Azure VMs have a patchMode setting that decides who patches the VM. Scheduled patching through AUM only works when this is set correctly.
Windows patchMode |
Linux patchMode |
Meaning |
|---|---|---|
AutomaticByPlatform |
AutomaticByPlatform |
Azure orchestrates. Combined with bypassPlatformSafetyChecksOnUserSchedule = true, this is "Customer Managed Schedules" — required for maintenance configurations. Without the bypass flag, Azure applies Automatic VM guest patching on its own safe-deployment cadence. |
AutomaticByOS |
— | Windows Update on the box decides (classic automatic updates). |
Manual |
— | Nothing automatic; you install manually or through WSUS/GPO. |
| — | ImageDefault |
Whatever the distro image does (e.g. unattended-upgrades on Ubuntu). |
For AUM schedules, set:
resource "azurerm_linux_virtual_machine" "app" {
# ...
patch_mode = "AutomaticByPlatform"
patch_assessment_mode = "AutomaticByPlatform"
bypass_platform_safety_checks_on_user_schedule_enabled = true
reboot_setting = "IfRequired" # only valid with AutomaticByPlatform
}
az vm update -g rg-app -n vm-app-01 \
--set osProfile.linuxConfiguration.patchSettings.patchMode=AutomaticByPlatform \
osProfile.linuxConfiguration.patchSettings.automaticByPlatformSettings.bypassPlatformSafetyChecksOnUserSchedule=true
Arc-enabled servers don't have patchMode; they're always "customer managed" as far as AUM is concerned.
Conflict to avoid: if a Linux VM is on AUM schedules and has unattended-upgrades or dnf-automatic installing on its own timer, you get two patching systems racing each other and nonsense compliance data. Pick one per server. A common split:
- AUM schedules for production (controlled windows, reporting, reboots coordinated).
- Native unattended tooling for low-risk dev/test or homelab, as covered in Unattended Updates.
Maintenance configurations and dynamic scopes¶
Design maintenance configurations around waves, not around individual servers. A typical monthly cycle aligned to Patch Tuesday (second Tuesday of the month):
| Configuration | Scope (dynamic) | Schedule | Content |
|---|---|---|---|
mc-nonprod |
env=dev,test |
Patch Tuesday + 1 day, 22:00 | Critical, Security, Updates |
mc-prod-wave1 |
patch-group=prod-wave1 (canaries) |
Patch Tuesday + 4 days, 02:00 | Critical, Security |
mc-prod-wave2 |
patch-group=prod-wave2 |
Patch Tuesday + 11 days, 02:00 | Critical, Security |
mc-prod-manual |
reboot-policy=manual |
None (one-time runs during a change) | — |
The recurrence syntax supports offsets from a weekday occurrence, so "four days after Patch Tuesday" is Month Second Tuesday Offset4.
Azure CLI¶
az maintenance configuration create \
--resource-group rg-patching \
--resource-name mc-prod-wave2 \
--location canadacentral \
--maintenance-scope InGuestPatch \
--extension-properties InGuestPatchMode="User" \
--maintenance-window-start-date-time "2026-12-01 02:00" \
--maintenance-window-duration "03:55" \
--maintenance-window-time-zone "Eastern Standard Time" \
--maintenance-window-recur-every "Month Second Tuesday Offset11" \
--reboot-setting IfRequired \
--install-patches-windows-parameters classifications-to-include="Critical" "Security" "UpdateRollup" \
--install-patches-linux-parameters classifications-to-include="Critical" "Security" package-name-masks-to-exclude="docker-ce*" "postgresql*"
Duration is the install window; AUM stops starting new installs near the end of it and reserves time for reboot. Windows needs at least 20 minutes reserved for reboot, Linux 15 — keep windows comfortably larger than your slowest server.
Terraform¶
resource "azurerm_maintenance_configuration" "prod_wave2" {
name = "mc-prod-wave2"
resource_group_name = azurerm_resource_group.patching.name
location = "canadacentral"
scope = "InGuestPatch"
in_guest_user_patch_mode = "User"
window {
start_date_time = "2026-12-01 02:00"
duration = "03:55"
time_zone = "Eastern Standard Time"
recur_every = "Month Second Tuesday Offset11"
}
install_patches {
reboot = "IfRequired"
windows {
classifications_to_include = ["Critical", "Security", "UpdateRollup"]
kb_numbers_to_exclude = []
}
linux {
classifications_to_include = ["Critical", "Security"]
package_names_mask_to_exclude = ["docker-ce*", "postgresql*"]
}
}
tags = { owner = "[email protected]" }
}
resource "azurerm_maintenance_assignment_dynamic_scope" "prod_wave2" {
name = "prod-wave2-scope"
maintenance_configuration_id = azurerm_maintenance_configuration.prod_wave2.id
filter {
locations = []
os_types = ["Windows", "Linux"]
resource_types = [
"Microsoft.Compute/virtualMachines",
"Microsoft.HybridCompute/machines",
]
tag_filter = "All"
tags {
tag = "patch-group"
values = ["prod-wave2"]
}
}
}
Dynamic scope assignments are made at subscription scope; create one per subscription that holds servers.
Exclusions that belong in the configuration, not in people's heads¶
- Database engines with their own upgrade process (PostgreSQL, MariaDB/MySQL, SQL Server CUs).
- Container runtimes on hosts where a restart takes down every container.
- Kernel packages on hosts where you manage kernels separately (livepatch, vendor-certified kernels).
- Agent software managed by its own updater (Defender for Endpoint, backup agents).
Every exclusion should map to an entry in the exception register with an owner and a reason, otherwise it becomes permanent by accident.
Pre- and post-maintenance events¶
Maintenance configurations publish events to an Event Grid system topic. Subscribe a webhook, Azure Function, or Automation runbook to:
| Event | Typical action |
|---|---|
Microsoft.Maintenance.PreMaintenanceEvent |
Start deallocated VMs (AUM skips stopped machines), take disk snapshots, put nodes into maintenance in the load balancer or monitoring, disable alerting noise |
Microsoft.Maintenance.PostMaintenanceEvent |
Run smoke tests, re-enable alerting, stop VMs that were started, post results to the team channel |
Pre-events have a short timeout before the schedule proceeds, so keep handlers fast and idempotent, and make them fail closed for critical tiers: if the snapshot step fails, cancel the run for that machine rather than patching without a rollback point.
Example post-event Function logic (pseudo-code):
def on_post_maintenance(event):
run_id = event["data"]["CorrelationId"]
results = resource_graph(f"""
patchinstallationresources
| where properties.maintenanceRunId =~ '{run_id}'
| project id, status = tostring(properties.status),
installed = toint(properties.installedPatchCount),
failed = toint(properties.failedPatchCount),
reboot = tostring(properties.rebootStatus)
""")
failed = [r for r in results if r["status"] != "Succeeded"]
post_to_channel(summary(results), alert=bool(failed))
for host in successful_hosts(results):
run_smoke_test(host)
Reporting with Azure Resource Graph¶
Resource Graph is the single query surface for both Azure VMs and Arc machines.
Machines with pending security or critical updates:
patchassessmentresources
| where type in~ (
"microsoft.compute/virtualmachines/patchassessmentresults",
"microsoft.hybridcompute/machines/patchassessmentresults")
| extend p = properties
| extend machineId = tostring(split(id, "/patchAssessmentResults/")[0])
| project machineId,
osType = tostring(p.osType),
lastAssessed = todatetime(p.lastModifiedDateTime),
critical = toint(p.availablePatchCountByClassification.critical),
security = toint(p.availablePatchCountByClassification.security),
rebootPending = tobool(p.rebootPending)
| where critical > 0 or security > 0
| order by critical desc, security desc
Machines whose assessment is stale (> 3 days):
patchassessmentresources
| where type endswith "/patchassessmentresults"
| extend lastAssessed = todatetime(properties.lastModifiedDateTime)
| where lastAssessed < ago(3d)
| project id, lastAssessed
Last installation result per machine:
patchinstallationresources
| where type endswith "/patchinstallationresults"
| extend p = properties
| summarize arg_max(todatetime(p.lastModifiedDateTime), *) by machine = tostring(split(id, "/patchInstallationResults/")[0])
| project machine, status = tostring(p.status), startedBy = tostring(p.startedBy),
installed = toint(p.installedPatchCount), failed = toint(p.failedPatchCount)
Servers with no maintenance configuration assigned:
resources
| where type in~ ("microsoft.compute/virtualmachines", "microsoft.hybridcompute/machines")
| join kind=leftouter (
maintenanceresources
| where type =~ "microsoft.maintenance/configurationassignments"
| extend machine = tolower(tostring(properties.resourceId))
| project machine
) on $left.id == $right.machine
| where isempty(machine)
| project id, name, tags
(Dynamic-scope matches don't always appear as per-machine assignment rows; validate this query against your own tenant and fall back to tag-based checks.)
Pin these to an Azure Workbook or export to your own dashboard. Property names in Resource Graph occasionally change — test queries in the portal's Resource Graph Explorer before wiring them into alerts.
Cost and licensing notes¶
- Azure VMs: Azure Update Manager is free.
- Arc-enabled servers: charged per server per day of management, except when the server is covered by Microsoft Defender for Servers Plan 2, is enrolled for Extended Security Updates through Arc, or runs on Azure Local with Azure benefits. Check the current pricing page — this has changed more than once.
- Machine configuration (guest configuration) on Arc: also billed per server unless covered by Defender for Servers / other benefits.
- Windows Server 2016 reaches end of extended support on January 12, 2027. ESUs are available through Azure Arc (generally available since August 2026). Unlike earlier ESU programs, don't assume ESU is free on Azure VMs for 2016 — confirm licensing before you plan around it, and treat ESU as a bridge to an upgrade, not a destination. See Patch and Upgrade Management.
- Hotpatch for Windows Server 2025 via Arc is available at no additional cost (since May 2026), but still needs a quarterly baseline reboot.
Hardening: choosing a baseline¶
Pick one primary baseline per OS family and document deviations. Mixing baselines without a source of truth produces endless arguments about which setting is "right."
| Baseline | Coverage | Strengths | Notes |
|---|---|---|---|
| CIS Benchmarks | Windows Server, RHEL/Rocky/Alma, Ubuntu, Debian, SLES, Amazon Linux, plus apps (SQL Server, PostgreSQL, Nginx, Tomcat, Kubernetes) | Industry standard; auditors recognise it; Level 1 (low impact) and Level 2 (defence in depth) profiles; separate Member Server / Domain Controller profiles | Free PDFs; CIS-CAT Pro, build kits and Group Policy objects need SecureSuite membership |
| Microsoft Security Baselines | Windows Server, Windows client, Edge, M365 apps | Shipped as GPO backups in the Security Compliance Toolkit; tested by Microsoft | Fewer settings than CIS; Windows only |
| Azure compute security baseline | Windows and Linux on Azure and Arc | Built into Azure Policy machine configuration; free audit on Azure VMs | Microsoft-defined, close to CIS but not identical |
| CIS baselines via Azure Policy (OSConfig) | Windows Server 2025 (audit-only initial release, May 2026), Linux distros | Native audit in Azure Policy without third-party tooling | Check coverage per OS before relying on it |
| DISA STIGs | Most enterprise OSes | Required in US government contexts | Strict; operational impact higher than CIS L1 |
Current CIS versions worth knowing as of late 2026: Windows Server 2025 Benchmark v2.1.0 and Windows Server 2022 v5.1.0 (both updated September 2026), Ubuntu 24.04 LTS v2.0.0, RHEL 9 v3.0.0, plus new Ubuntu 26.04 coverage. Benchmarks update several times a year — record the exact version you implement.
Profile selection rule of thumb: CIS Level 1 everywhere; Level 2 on tier-0 systems (domain controllers, PKI, jump hosts, anything holding secrets for other systems) where you can test thoroughly.
Hardening Windows Server¶
Domain-joined servers¶
Group Policy remains the most reliable enforcement mechanism for domain members.
- Import the baseline GPOs (CIS build kit or Microsoft Security Compliance Toolkit) into a test OU first.
- Split into layered GPOs rather than one monolith:
-
SEC-Baseline-MemberServer(CIS L1 Member Server) -SEC-Baseline-DomainController-SEC-Role-IIS,SEC-Role-SQL,SEC-Role-FileServer(role-specific relaxations or additions) -SEC-Exceptions-<app>(documented deviations, linked to exception IDs) - Use
LocalGPO/LGPO.exefrom the toolkit for workgroup or non-domain servers.
High-value settings that frequently show up as findings:
# SMBv1 off
Disable-WindowsOptionalFeature -Online -FeatureName SMB1Protocol -NoRestart
Set-SmbServerConfiguration -EnableSMB1Protocol $false -Force
# SMB signing required (server and client)
Set-SmbServerConfiguration -RequireSecuritySignature $true -Force
Set-SmbClientConfiguration -RequireSecuritySignature $true -Force
# Disable legacy TLS/SSL protocols in SChannel
foreach ($proto in 'SSL 2.0','SSL 3.0','TLS 1.0','TLS 1.1') {
foreach ($side in 'Server','Client') {
$k = "HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\$proto\$side"
New-Item -Path $k -Force | Out-Null
New-ItemProperty -Path $k -Name Enabled -Value 0 -PropertyType DWord -Force | Out-Null
New-ItemProperty -Path $k -Name DisabledByDefault -Value 1 -PropertyType DWord -Force | Out-Null
}
}
# LSA protection (RunAsPPL)
New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Lsa' -Name RunAsPPL -Value 1 -PropertyType DWord -Force
# NTLMv2 only, refuse LM & NTLM
New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Lsa' -Name LmCompatibilityLevel -Value 5 -PropertyType DWord -Force
# Print Spooler off where not needed (CIS recommends disabling on DCs and most servers)
Set-Service -Name Spooler -StartupType Disabled; Stop-Service -Name Spooler -Force
Treat these snippets as illustrations — in production, set them through GPO or DSC so they're enforced continuously, not once.
Additional controls that baselines assume but don't configure:
- LAPS (Windows LAPS is built in from the April 2023 updates onward) for unique local admin passwords.
- Credential Guard and HVCI on Windows Server 2022/2025 where hardware supports it (watch for driver compatibility).
- Defender for Endpoint with tamper protection; ASR rules in audit first.
- Windows Firewall enabled on all profiles with inbound default deny.
Validation¶
Get-SmbServerConfiguration,Get-TlsCipherSuite,auditpol /get /category:*for spot checks.Microsoft Policy Analyzer(toolkit) to diff effective GPO against baseline.- CIS-CAT Pro or your scanner's compliance module (Qualys Policy Compliance, Tenable audit files, Defender Vulnerability Management security baselines assessment) for scored reports.
Hardening Linux¶
Tooling per distro¶
| Distro | Native/vendor tooling | Community automation |
|---|---|---|
| RHEL / Rocky / Alma / Oracle | scap-security-guide + oscap (CIS and STIG profiles), RHEL Image Builder with OpenSCAP remediation at build time |
ansible-lockdown/RHEL9-CIS, RHEL8-CIS |
| Ubuntu | Ubuntu Security Guide (usg) with Ubuntu Pro — CIS and DISA-STIG profiles, audit and fix |
ansible-lockdown/UBUNTU24-CIS, UBUNTU22-CIS |
| Debian | OpenSCAP content is limited; mostly community roles | ansible-lockdown roles, dev-sec/ansible-collection-hardening |
| SLES | scap-security-guide profiles |
community roles |
| Azure Linux | Microsoft-hardened by default; check machine configuration CIS policies | — |
OpenSCAP on RHEL-family¶
sudo dnf install -y openscap-scanner scap-security-guide
ls /usr/share/xml/scap/ssg/content/ # ssg-rhel9-ds.xml, etc.
# List profiles
oscap info /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml | grep -A1 'Profile'
# Audit against CIS Level 1 Server
sudo oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_cis_server_l1 \
--results /var/tmp/cis-results.xml \
--report /var/tmp/cis-report.html \
/usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml
# Generate an Ansible playbook that remediates only the failed rules
oscap xccdf generate fix --fix-type ansible \
--result-id "" /var/tmp/cis-results.xml > cis-remediate.yml
Review generated remediation before running it — it applies everything that failed, including settings your applications may depend on.
Ubuntu Security Guide¶
sudo pro attach <token>
sudo pro enable usg
sudo apt install -y usg
sudo usg audit cis_level1_server # HTML + XML report in /var/lib/usg
sudo usg generate-tailoring cis_level1_server tailor.xml # customise rules
sudo usg audit --tailoring-file tailor.xml
sudo usg fix --tailoring-file tailor.xml # apply (test first!)
Tailoring files are the right place to record deviations; store them in Git next to the exception register.
Settings that most often break things¶
| Control | Typical breakage | Mitigation |
|---|---|---|
Separate /tmp, /var/tmp with noexec |
Installers and some apps execute from /tmp |
Set TMPDIR for those apps; exception for build hosts |
Disable unused filesystems (cramfs, squashfs, udf) |
Snap packages need squashfs |
Exclude rule on Ubuntu if snaps are used |
SSH AllowGroups/AllowUsers |
Automation accounts locked out | Add automation group before rollout |
auditd rules with -e 2 (immutable) |
Changes to audit rules require reboot | Fine for prod; leave mutable in dev |
| Firewall enabled with default deny | Monitoring/backup agents blocked | Inventory listening ports first (ss -tlnp) |
kernel.yama.ptrace_scope |
Debuggers, some APM agents | Exception per host |
| Password/PAM policy | Breaks LDAP/SSSD auth stacks if applied blindly | Test against your auth setup |
Auditing baselines at scale with machine configuration¶
Azure Policy machine configuration (formerly guest configuration) audits — and optionally enforces — settings inside the OS of Azure VMs and Arc machines, and shows results per setting in the Policy compliance blade.
Built-in initiatives to start with:
- Windows machines should meet requirements of the Azure compute security baseline
- Linux machines should meet requirements for the Azure compute security baseline
- CIS-based machine configuration policies for supported Linux distros and Windows Server 2025 (check the built-in definitions list for your OS versions)
Assignment pattern:
- Assign the prerequisite initiative Deploy prerequisites to enable Guest Configuration policies on virtual machines (adds system-assigned identity and the extension).
- Assign the baseline initiative in Audit mode at management group scope.
- Export non-compliant settings, compare against your chosen CIS profile, and record deviations as exemptions (see Exception Management).
- For a subset of settings you're confident about, author custom packages with
ApplyAndAutoCorrect(see Configuration Management).
Query results via Resource Graph:
guestconfigurationresources
| where name contains "AzureLinuxBaseline" or name contains "AzureWindowsBaseline"
| extend compliance = tostring(properties.complianceStatus),
machine = tostring(split(properties.targetResourceId, "/")[8])
| project machine, compliance, lastChecked = todatetime(properties.lastComplianceStatusChecked)
Defender for Cloud also surfaces OS misconfigurations through Defender Vulnerability Management security baseline assessments (Defender for Servers Plan 2), mapped to CIS. Use whichever source your compliance reporting is built on, but don't run three overlapping audits and reconcile them by hand.
Rolling hardening out without breaking things¶
- Inventory dependencies — listening ports, service accounts, scheduled tasks/cron, agents, auth methods.
- Audit only across the whole estate. Produce a per-setting failure count.
- Group settings by risk: zero-impact (banner text, log retention), low-impact (SMB signing, TLS 1.0 off), high-impact (firewall default deny, PAM changes,
noexecmounts). - Pilot on non-production servers that mirror production roles. Run application test suites, not just "it booted."
- Canary production — one server per role, monitor for a full business cycle (including month-end jobs if relevant).
- Wave rollout reusing the patch groups.
- Enforce continuously (GPO, DSC, Ansible, machine configuration auto-correct) and alert on drift.
- Document every deviation as an exception with an expiry date.
Bake hardening into images too: build golden images (Packer, Azure Image Builder, RHEL Image Builder) that already pass the baseline, so new servers start compliant rather than being fixed after deployment.
Servers that can't be Arc-enabled¶
Some servers will never run the Arc agent: air-gapped networks, appliances, vendor-locked systems, customer-owned hosts in a hosting contract.
- Patch: WSUS/Configuration Manager for Windows; a local repository mirror (Foreman/Katello, Pulp,
apt-mirror/aptly, Satellite) plus Ansible for Linux. - Assess: authenticated scans from Qualys/Tenable scanner appliances inside the network segment.
- Harden: Ansible or GPO; CIS-CAT or OpenSCAP reports exported for evidence.
- Report: push results into the same dashboard (Log Analytics custom table via the Logs Ingestion API, or your SIEM) so these servers aren't invisible.
Record why each is excluded from Arc, with an owner — it's an exception like any other.
Operational checklist¶
Daily - [ ] Review failed or partially failed patch runs from last night - [ ] Check machines with stale assessments (> 3 days) - [ ] Triage new critical/KEV findings (see Vulnerability Remediation)
Weekly - [ ] Reconcile Arc inventory vs CMDB vs scanner - [ ] Review Arc agents disconnected > 7 days - [ ] Review reboot-pending machines
Monthly
- [ ] Patch Tuesday wave plan confirmed and communicated
- [ ] Baseline compliance trend reviewed; new deviations recorded as exceptions
- [ ] Arc agent version currency (azcmagent auto-upgrade or update through your package manager)
Quarterly - [ ] Re-validate baseline version against latest CIS release - [ ] Review all hardening exceptions for expiry - [ ] EOL horizon review (next 12 months) - [ ] Hotpatch baseline reboot months scheduled (Windows Server 2025 hotpatch)
References¶
- Azure Update Manager documentation — Microsoft Learn
- Azure Update Manager pricing — azure.microsoft.com/pricing/details/azure-update-management-center
- Azure Arc-enabled servers — Microsoft Learn
- Azure Policy machine configuration — Microsoft Learn
- CIS Benchmarks — cisecurity.org/cis-benchmarks
- Microsoft Security Compliance Toolkit — Microsoft Download Center
- Ubuntu Security Guide — ubuntu.com/security/certifications/docs/usg
- ComplianceAsCode / scap-security-guide — github.com/ComplianceAsCode/content
- Ansible Lockdown — github.com/ansible-lockdown