Network Changes and Runbooks¶
Many findings are fixed in the network, not on the server: an exposed management port, a storage account reachable from the internet, a PaaS service that should only be reachable privately. Those changes are usually owned by a networking team, so remediation becomes a coordination problem. And every procedure you repeat — patching, emergency remediation, network validation — needs a runbook that someone else can follow at 03:00.
Contents¶
Networking
- Common remediation-driven network changes
- NSG essentials
- Closing exposed management ports
- Private endpoints
- Private DNS: where private endpoints go wrong
- VNet and egress changes
- Network paths that patching and scanning depend on
- Writing a network change request the network team will love
- Validating network changes
- Flow logs and evidence
Documentation
- Runbooks
- Runbook template
- Example: monthly patching runbook
- Example: emergency vulnerability remediation runbook
- Example: make a PaaS service private runbook
- Keeping documentation alive
Common remediation-driven network changes¶
| Finding | Network change | Typical owner |
|---|---|---|
RDP/SSH/WinRM open to 0.0.0.0/0 or Internet |
Remove rule; use Azure Bastion, JIT VM access, or VPN; restrict to admin subnets | Network + server owner |
| Storage account / Key Vault / SQL / ACR publicly accessible | Private endpoint + disable public network access (or firewall to selected networks) | Network + resource owner |
| App Service publicly reachable but only used internally | Private endpoint for inbound; VNet integration for outbound; access restrictions | App platform + network |
| Management plane of AKS public | Private cluster or authorized IP ranges / API server VNet integration | Platform + network |
| Unrestricted outbound internet from servers | Route 0.0.0.0/0 via Azure Firewall / NVA; NAT gateway for controlled egress; FQDN allow lists |
Network |
| Database ports reachable from broad ranges | Tighten NSG to app subnets / ASGs | Network + DB owner |
| Legacy protocols on network (SMBv1, Telnet, FTP) between segments | Block at NSG/firewall while servers are fixed | Network |
| No flow visibility for incident investigation | VNet flow logs + traffic analytics | Network |
NSG essentials¶
Enough to have a productive conversation with the network team:
- Rules are evaluated by priority (100–4096, lower first); first match wins.
- Default rules (priority 65000+) allow VNet-to-VNet and Azure Load Balancer inbound, and deny everything else inbound; outbound defaults allow VNet and Internet.
- NSGs attach to subnets and/or NICs; for inbound traffic the subnet NSG is evaluated first, then the NIC NSG (reverse for outbound). Both must allow.
- Service tags (
AzureCloud,Storage.CanadaCentral,AzureMonitor,AzureArcInfrastructure,GuestAndHybridManagement,VirtualNetwork,Internet) are Microsoft-maintained IP sets — prefer them over hard-coded IP ranges. - Application security groups (ASGs) group NICs by role (
asg-web,asg-sql) so rules read like intent: allowasg-web→asg-sqlon 1433. - Augmented rules allow multiple prefixes/ports per rule, reducing rule sprawl.
Rule naming convention that helps audits:
<direction>-<allow|deny>-<source>-to-<destination>-<port/proto>-<ticket>
e.g. in-allow-asg-web-to-asg-sql-1433-tcp-CHG12345
Closing exposed management ports¶
Find them¶
// NSG rules allowing inbound management ports from the internet
resources
| where type =~ "microsoft.network/networksecuritygroups"
| mv-expand rule = properties.securityRules
| extend r = rule.properties
| where r.direction == "Inbound" and r.access == "Allow"
| extend src = tostring(r.sourceAddressPrefix), srcs = r.sourceAddressPrefixes,
port = tostring(r.destinationPortRange), ports = r.destinationPortRanges
| where src in ("*", "0.0.0.0/0", "Internet", "Any") or set_has_element(srcs, "Internet") or set_has_element(srcs, "0.0.0.0/0")
| where port in ("22", "3389", "5985", "5986", "*") or set_has_element(ports, "22") or set_has_element(ports, "3389")
| project nsg = name, resourceGroup, rule = tostring(rule.name), src, port, priority = toint(r.priority)
Defender for Cloud also raises Management ports should be closed on your virtual machines and Internet-facing virtual machines should be protected with network security groups.
Replace access paths before removing rules¶
- Azure Bastion (Standard/Premium for native client and IP-based connections to on-prem/peered hosts) — browser or
az network bastion ssh/rdp. - Just-in-time VM access (Defender for Servers) — opens the port to a requester's IP for a limited time with approval and audit.
- VPN / ExpressRoute + jump hosts / PAWs for admins.
- Arc SSH access (
az ssh arc) for Arc-enabled Linux and Windows servers — no inbound port needed on the server.
Then remove the rule, verify nothing broke (monitoring, backup, automation accounts often use the same port), and close the finding.
Private endpoints¶
A private endpoint puts a NIC with a private IP from your VNet in front of a PaaS resource. With public network access disabled, the resource is reachable only through that IP.
Terraform example: Key Vault¶
resource "azurerm_key_vault" "orders" {
name = "kv-orders-prod"
location = var.location
resource_group_name = var.rg
tenant_id = var.tenant_id
sku_name = "standard"
enable_rbac_authorization = true
public_network_access_enabled = false
purge_protection_enabled = true
network_acls {
default_action = "Deny"
bypass = "AzureServices"
}
}
resource "azurerm_private_endpoint" "kv_orders" {
name = "pe-kv-orders-prod"
location = var.location
resource_group_name = var.rg
subnet_id = data.azurerm_subnet.private_endpoints.id
private_service_connection {
name = "psc-kv-orders-prod"
private_connection_resource_id = azurerm_key_vault.orders.id
subresource_names = ["vault"]
is_manual_connection = false
}
private_dns_zone_group {
name = "default"
private_dns_zone_ids = [data.azurerm_private_dns_zone.vault.id] # privatelink.vaultcore.azure.net
}
}
Sub-resource names and zones you'll use most¶
| Service | subresource_names |
Private DNS zone |
|---|---|---|
| Key Vault | vault |
privatelink.vaultcore.azure.net |
| Storage (blob) | blob |
privatelink.blob.core.windows.net |
| Storage (file) | file |
privatelink.file.core.windows.net |
| Azure SQL | sqlServer |
privatelink.database.windows.net |
| ACR | registry |
privatelink.azurecr.io (plus regional data endpoints) |
| App Service (inbound) | sites |
privatelink.azurewebsites.net |
| Service Bus / Event Hubs | namespace |
privatelink.servicebus.windows.net |
| Azure Monitor (AMPLS) | azuremonitor |
several (privatelink.monitor.azure.com, privatelink.oms.opinsights.azure.com, …) |
| Azure Arc (Private Link Scope) | hybridcompute |
privatelink.his.arc.azure.com, privatelink.guestconfiguration.azure.com, privatelink.dp.kubernetesconfiguration.azure.com |
Order of operations (avoid self-inflicted outages)¶
- Create the private endpoint and DNS records while public access is still enabled.
- Verify every consumer resolves the name to the private IP and connects successfully (including on-prem consumers, build agents, Functions, App Services via VNet integration).
- Check the resource's own logs for remaining public-IP clients.
- Then disable public network access.
- Keep a rollback: re-enabling public access is one setting — document it in the runbook.
NSGs on private endpoints¶
Network policies (NSG and UDR) for private endpoints are off by default on a subnet. To filter traffic to private endpoints with NSGs, enable them on the subnet:
az network vnet subnet update -g rg-net --vnet-name vnet-hub -n snet-private-endpoints \
--private-endpoint-network-policies Enabled
Private DNS: where private endpoints go wrong¶
Almost every "private endpoint doesn't work" ticket is DNS.
How it should resolve:
kv-orders-prod.vault.azure.net
→ CNAME kv-orders-prod.privatelink.vaultcore.azure.net
→ (inside your network, private DNS zone) A 10.20.30.4
→ (on the internet) public IP (blocked when public access disabled)
Patterns:
| Consumer | How it resolves privately |
|---|---|
| Azure VMs/App Services in linked VNets | Private DNS zone linked to the VNet (Azure-provided DNS 168.63.129.16) |
| Hub-spoke with custom DNS servers | Custom DNS forwards privatelink.* (or the public zones) to Azure DNS Private Resolver inbound endpoint or to 168.63.129.16 from a VM in a linked VNet |
| On-premises clients | Conditional forwarders for the public zone names (e.g. vault.azure.net, blob.core.windows.net) → Private Resolver inbound endpoint |
Forward the public zone (e.g. blob.core.windows.net), not the privatelink zone, from on-prem — clients query the public name and follow the CNAME.
Testing:
nslookup kv-orders-prod.vault.azure.net # expect privatelink CNAME → 10.x
dig +short kv-orders-prod.vault.azure.net
Resolve-DnsName kv-orders-prod.vault.azure.net
Test-NetConnection kv-orders-prod.vault.azure.net -Port 443
Centralise private DNS zones in the hub (one zone per service type, linked to all spokes) and let Azure Policy create the privateDnsZoneGroup automatically when private endpoints are deployed — otherwise every team creates its own zone and resolution becomes inconsistent.
VNet and egress changes¶
Remediation sometimes needs bigger moves:
- VNet integration for App Service/Functions so outbound calls reach private endpoints (
az webapp vnet-integration add), with route-all enabled if egress must traverse the firewall. - Forced tunnelling / UDRs sending
0.0.0.0/0to Azure Firewall — check that platform dependencies still work (AKS needs specific FQDNs and service tags; Windows activation needs KMS endpoints; Arc and Update Manager need their endpoints). - NAT gateway for predictable egress IPs (allow-listing at partners) and to remove default outbound access, which Azure is retiring for new VNets in favour of explicit outbound methods.
- Subnet re-addressing / new subnets for private endpoints or delegated services — plan IP space with the network team early; this is often the long pole.
Network paths that patching and scanning depend on¶
When tightening egress, keep these working (verify current endpoint lists in Microsoft docs; they change):
| Need | Allow |
|---|---|
| Azure Arc agent | Service tags AzureArcInfrastructure, AzureResourceManager, AzureActiveDirectory, GuestAndHybridManagement (or Arc Private Link Scope) |
| Azure Update Manager on Azure VMs | Windows Update / Microsoft Update endpoints, or your WSUS; Linux distro repos or your mirror (*.ubuntu.com, RHUI for Azure RHEL images, etc.) |
| Defender for Endpoint / Defender for Servers | MDE endpoints (streamlined connectivity list) |
| Azure Monitor Agent | AzureMonitor service tag or AMPLS |
| Vulnerability scanners (Qualys/Tenable) | Agent → cloud platform URLs; scanner appliance → targets on the scanned ports |
| Container builds / AKS | ACR, MCR (mcr.microsoft.com, *.data.mcr.microsoft.com), package registries used in builds |
A firewall change that quietly blocks the update path produces "compliant" servers that are simply no longer checking. Pair egress changes with the stale-assessment alert from Pipeline Scanning and Monitoring.
Writing a network change request the network team will love¶
Network teams receive vague requests ("open the firewall for the app") all day. A precise request gets done faster.
Title: Allow web tier → orders SQL on 1433 (replace broad 10.0.0.0/8 rule) — remediation VM-2026-00987
Driver:
Finding VM-2026-00987 (P3): SQL port 1433 reachable from 10.0.0.0/8 on nsg-sql-prod.
Target state: only web tier may reach SQL.
Change:
ADD nsg-sql-prod in-allow-asg-web-to-asg-sql-1433-tcp-CHG12345
priority 200, src ASG asg-web-prod, dst ASG asg-sql-prod, TCP 1433, Allow
REMOVE nsg-sql-prod in-allow-10.0.0.0-8-to-any-1433 (priority 300)
Who/what uses the path today (from VNet flow logs, last 30 days):
10.20.1.0/24 (web VMs) — expected, covered by new rule
10.30.5.17 (backup server) — uses 1433 for SQL backups → ADD rule for asg-backup
10.40.2.9 (unknown) — investigating, owner TBD; will not be covered
Validation:
- Network Watcher IP flow verify: web → sql:1433 Allow; 10.40.2.9 → sql:1433 Deny
- App smoke test: orders checkout end-to-end
- Backup job succeeds next run
Rollback:
Re-add previous rule (exported JSON attached), priority 300.
Window: <date/time>, standard change
Contacts: requester (remediation owner), app owner, DB owner
The single most valuable section is "who uses the path today" — pulled from flow logs. It turns a risky change into a known one.
Validating network changes¶
Before and after:
# Would this flow be allowed? (evaluates effective NSG rules on the NIC)
az network watcher test-ip-flow \
--vm vm-web-01 -g rg-web --direction Outbound --protocol TCP \
--local 10.20.1.4:50000 --remote 10.20.2.5:1433
# Effective security rules on a NIC (combined subnet + NIC NSGs)
az network nic list-effective-nsg -g rg-web -n vm-web-01-nic
# Effective routes (did a UDR send traffic to the firewall?)
az network nic show-effective-route-table -g rg-web -n vm-web-01-nic -o table
# End-to-end connectivity check from a VM (uses Network Watcher agent extension)
az network watcher test-connectivity -g rg-web --source-resource vm-web-01 \
--dest-address kv-orders-prod.vault.azure.net --dest-port 443
From the hosts themselves:
nc -vz 10.20.2.5 1433
curl -sv https://kv-orders-prod.vault.azure.net/healthstatus -o /dev/null
Test-NetConnection 10.20.2.5 -Port 1433
Capture outputs in the change record — they're the closure evidence for the finding.
Flow logs and evidence¶
- VNet flow logs replace NSG flow logs: NSG flow logs can no longer be created (since June 30, 2025) and are retired on September 30, 2027, when existing ones are deleted. Migrate now.
- Enable traffic analytics on VNet flow logs for a queryable view (
NTANetAnalyticstable) of who talks to whom. - Use them before changes (who uses this path?) and after (did denied traffic appear that we didn't expect?).
NTANetAnalytics
| where TimeGenerated > ago(30d)
| where DestPort == 1433 and DestIp == "10.20.2.5"
| summarize flows = sum(AllowedInFlows + DeniedInFlows), lastSeen = max(TimeGenerated) by SrcIp
| order by flows desc
(Column names in traffic analytics tables can differ by schema version; run getschema to confirm.)
Runbooks¶
A runbook is a procedure written so that someone who didn't write it can execute it correctly, under pressure, and know when to stop.
Qualities of a good runbook¶
| Quality | What it looks like |
|---|---|
| Actionable | Numbered steps with exact commands, not prose descriptions |
| Verifiable | Every step that changes something has an expected result and a check |
| Safe | Pre-checks, stop conditions, rollback section — before the steps start |
| Scoped | Says what it covers and what it doesn't |
| Owned | Named owner and last-reviewed date |
| Linked | Alerts and tickets link to it; it links to the systems it touches |
| Tested | Executed recently (game days, or literally the last patch cycle) |
Types you need for this program¶
- Monthly patching (per platform / per wave)
- Emergency (P1/KEV) vulnerability remediation
- Out-of-band patch deployment
- Rollback procedures (snapshot restore, KB uninstall, package downgrade, slot swap,
kubectl rollout undo) - Arc onboarding and troubleshooting
- Making a PaaS service private (private endpoint migration)
- Closing exposed management ports
- Base image rebuild and redeploy
- Exception request and review
- Secret leak response (rotate → investigate → replace pattern)
Runbook template¶
---
title: "<Runbook name>"
date: YYYY-MM-DD
lastmod: YYYY-MM-DD
description: "<one line: what this does and when to use it>"
categories: ["vulnerability-management"]
tags: ["runbook", "<platform>", "<topic>"]
draft: false
---
# <Runbook name>
| | |
|---|---|
| **Purpose** | |
| **When to use** | Trigger(s): alert name, ticket type, schedule |
| **Scope** | Systems/environments covered — and explicitly not covered |
| **Owner** | Person/team |
| **Last reviewed / last executed** | YYYY-MM-DD / YYYY-MM-DD |
| **Estimated duration** | |
| **Risk** | Low / Medium / High — what could go wrong |
| **Change type** | Standard / Normal / Emergency |
## Prerequisites
- Access/roles required
- Tools (versions)
- Approvals
## Pre-checks
1. <check> — **Expected:** <result>. **If not:** stop / do X.
## Procedure
1. <step>
```bash
<exact command>
```
**Expected:** <output/result>
**If not:** <what to do>
## Verification
- <how to prove success>
## Rollback
- **Trigger:** <conditions>
- **Steps:**
1.
## Communication
- Before / during / after: who, which template
## Escalation
| Situation | Contact |
|---|---|
## References
- Related runbooks, docs, dashboards
## Change log
| Date | Change | By |
|---|---|---|
Example: monthly patching runbook¶
# Monthly Patching — Production Wave 2 (Azure + Arc)
| | |
|---|---|
| Purpose | Apply monthly security/critical updates to prod-wave2 servers |
| When to use | Patch Tuesday + 11 days, per change CHG-standard-PATCH-W2 |
| Scope | Servers tagged patch-group=prod-wave2 (Azure VMs and Arc). NOT clusters (see Wave 4 runbook) |
| Owner | Platform team |
| Duration | 4 hours window; ~30 min hands-on |
| Risk | Medium — reboots of non-redundant services listed in the pre-notice |
## Pre-checks (T-24h)
1. Ring 2 canaries patched with no open incidents. **Expected:** canary dashboard green.
2. Microsoft release health + distro errata reviewed for known issues. Record decisions in the change.
3. Pre-notice sent (template in Patch and Upgrade Management guide).
4. Resource Graph: confirm scope count matches expectations.
```kusto
resources
| where type in~ ("microsoft.compute/virtualmachines","microsoft.hybridcompute/machines")
| where tags["patch-group"] == "prod-wave2"
| summarize count() by type, tostring(properties.osType)
```
**If count differs by > 5% from last month:** investigate before proceeding.
## Pre-checks (T-30m)
5. Pre-maintenance event handler healthy (last run succeeded). Snapshots enabled for tier-1.
6. Monitoring maintenance mode scheduled for the window.
## Procedure
7. Window starts automatically (maintenance configuration mc-prod-wave2). Post "[STARTED]" message.
8. At T+60m: check progress.
```kusto
patchinstallationresources
| where properties.lastModifiedDateTime > ago(2h)
| summarize count() by tostring(properties.status)
```
**Expected:** mostly InProgress/Succeeded. **If any Failed:** open the machine's history; triage (disk space, pending reboot, repo reachability).
9. At window end: post "[COMPLETED]" summary with counts and failures.
## Verification
10. Smoke tests from post-event: all passing.
11. Trigger on-demand assessment for failed/partial machines; schedule one-time updates in the catch-up window.
12. Dashboard: no machines in scope with pending security updates (except documented exceptions).
## Rollback
- Trigger: service health failing > 15 min after reboot with no quick fix, or app owner declares critical breakage.
- Steps: see Rollback runbook (snapshot restore for Azure VMs; KB/package rollback per platform).
- After rollback: raise exception for the unapplied update; post "[ISSUE]" update.
## Escalation
| Situation | Contact |
|---|---|
| > 5% install failures | Platform lead |
| Customer-facing impact | Incident manager (major incident process) |
Example: emergency vulnerability remediation runbook¶
# Emergency Remediation — Actively Exploited Vulnerability (P1 / KEV)
| | |
|---|---|
| When to use | CVE added to CISA KEV affecting our estate, vendor emergency advisory, or exploitation observed |
| Change type | Emergency |
| Target | Mitigated within hours; remediated within P1 SLA |
## 1. Scope (first 60 minutes)
1. Identify affected assets:
- Servers: Defender for Cloud / MDVM software inventory search for product+version; Resource Graph
- Images: registry/runtime findings for the CVE; `kubectl get pods -A -o jsonpath=...` for image refs
- Network-only assets: Qualys/Tenable search by QID/plugin
2. Determine exposure per asset: internet-facing? reachable path? (NSG/Front Door/App Gateway config)
3. Open an incident/emergency change; page asset owners of exposed systems.
## 2. Mitigate (if a patch can't be deployed within hours)
4. Apply vendor mitigation (config change / feature disable) OR
5. Network block: NSG deny rule / WAF custom rule / disable public access. Record rule names for removal later.
6. Confirm mitigation with a test (vendor-provided check, scanner, or reproduction attempt in non-prod).
## 3. Remediate
7. Test patch on one non-prod system of each affected type.
8. Deploy: internet-facing first, then internal; use one-time updates / image rebuild + redeploy.
9. Reboot/restart as required; confirm running version.
## 4. Verify and hunt
10. On-demand assessment / re-scan; attach evidence to each finding.
11. Threat hunt for prior exploitation (EDR advanced hunting, web/app logs, sign-in logs) for the exposure period.
12. If indicators found → hand to incident response; preserve evidence.
## 5. Close
13. Remove temporary mitigations that are no longer needed (or convert to permanent controls via change).
14. Communicate completion to stakeholders.
15. Post-incident review within 5 business days: time to detect, scope, mitigate, remediate; what slowed us down.
Example: make a PaaS service private runbook¶
# Make a PaaS Resource Private (Private Endpoint Migration)
| | |
|---|---|
| When to use | Finding: "<service> should disable public network access" / "should use private link" |
| Scope | Key Vault, Storage, SQL, ACR, App Service (inbound), Service Bus/Event Hubs |
| Risk | High if consumers aren't inventoried — public access disable is the breaking step |
## Pre-checks
1. Inventory consumers: resource diagnostic logs (caller IPs) for 14–30 days; app config search; ask app owner.
**Expected:** every caller identified as in-VNet, on-prem via ExpressRoute/VPN, or an Azure service with trusted-service bypass.
2. Confirm private DNS zone for the service exists in the hub and is linked to consumer VNets; on-prem forwarders configured.
3. Confirm subnet for private endpoints has free IPs.
## Procedure
4. Create private endpoint (Terraform PR) with DNS zone group. Apply.
5. From each consumer network: `nslookup <name>` → expect private IP. `Test-NetConnection <name> -Port 443` → success.
6. Monitor resource logs for 24–72 h: remaining public-IP callers?
**If any:** identify and migrate them before continuing.
7. Disable public network access (Terraform PR: public_network_access_enabled = false).
8. Run consumer smoke tests.
## Verification
9. Defender recommendation for the resource turns healthy.
10. External test from internet: connection refused/forbidden.
## Rollback
- Re-enable public network access (single setting); leaves PE in place. Re-test consumers.
Keeping documentation alive¶
Documentation decays at the rate the systems change. Treat it like code:
- Docs as code: Markdown in Git, merge requests for changes, review by someone who'll use it.
- Front matter with
lastmodand an explicit Last reviewed / last executed line; a CI job lists runbooks not reviewed in 6 months. - Update in the same change: if a change alters a procedure, the runbook update is part of the change's definition of done.
- Link from alerts: every alert's description/annotation includes
runbook_url. Unused runbooks are a signal; alerts with no runbook are a gap. - Execute to review: the best review is following the runbook literally during the next patch cycle or a game day, noting every place reality differed.
- Retro feedback: after incidents and failed windows, update the runbook before closing the post-incident action items.
- Link-check: run a link checker over the docs site in CI so references to moved docs and portals don't rot.
A runbook nobody has run in a year is a hypothesis. Schedule the tests.
References¶
- Azure network security groups overview — Microsoft Learn
- Azure Private Link and private endpoint DNS configuration — Microsoft Learn
- Azure DNS Private Resolver — Microsoft Learn
- Network Watcher (IP flow verify, effective security rules, connection troubleshoot) — Microsoft Learn
- VNet flow logs and NSG flow logs retirement (September 30, 2027) — Microsoft Learn
- Azure Bastion and Just-in-time VM access — Microsoft Learn
- Google SRE Workbook, "On-Call" and "Incident Response" chapters (runbook practices)