Andrew Mercer
on this page

Network Changes and Runbooks

Many findings are fixed in the network, not on the server: an exposed management port, a storage account reachable from the internet, a PaaS service that should only be reachable privately. Those changes are usually owned by a networking team, so remediation becomes a coordination problem. And every procedure you repeat — patching, emergency remediation, network validation — needs a runbook that someone else can follow at 03:00.


Contents

Networking

  1. Common remediation-driven network changes
  2. NSG essentials
  3. Closing exposed management ports
  4. Private endpoints
  5. Private DNS: where private endpoints go wrong
  6. VNet and egress changes
  7. Network paths that patching and scanning depend on
  8. Writing a network change request the network team will love
  9. Validating network changes
  10. Flow logs and evidence

Documentation

  1. Runbooks
  2. Runbook template
  3. Example: monthly patching runbook
  4. Example: emergency vulnerability remediation runbook
  5. Example: make a PaaS service private runbook
  6. Keeping documentation alive

Common remediation-driven network changes

Finding Network change Typical owner
RDP/SSH/WinRM open to 0.0.0.0/0 or Internet Remove rule; use Azure Bastion, JIT VM access, or VPN; restrict to admin subnets Network + server owner
Storage account / Key Vault / SQL / ACR publicly accessible Private endpoint + disable public network access (or firewall to selected networks) Network + resource owner
App Service publicly reachable but only used internally Private endpoint for inbound; VNet integration for outbound; access restrictions App platform + network
Management plane of AKS public Private cluster or authorized IP ranges / API server VNet integration Platform + network
Unrestricted outbound internet from servers Route 0.0.0.0/0 via Azure Firewall / NVA; NAT gateway for controlled egress; FQDN allow lists Network
Database ports reachable from broad ranges Tighten NSG to app subnets / ASGs Network + DB owner
Legacy protocols on network (SMBv1, Telnet, FTP) between segments Block at NSG/firewall while servers are fixed Network
No flow visibility for incident investigation VNet flow logs + traffic analytics Network

NSG essentials

Enough to have a productive conversation with the network team:

  • Rules are evaluated by priority (100–4096, lower first); first match wins.
  • Default rules (priority 65000+) allow VNet-to-VNet and Azure Load Balancer inbound, and deny everything else inbound; outbound defaults allow VNet and Internet.
  • NSGs attach to subnets and/or NICs; for inbound traffic the subnet NSG is evaluated first, then the NIC NSG (reverse for outbound). Both must allow.
  • Service tags (AzureCloud, Storage.CanadaCentral, AzureMonitor, AzureArcInfrastructure, GuestAndHybridManagement, VirtualNetwork, Internet) are Microsoft-maintained IP sets — prefer them over hard-coded IP ranges.
  • Application security groups (ASGs) group NICs by role (asg-web, asg-sql) so rules read like intent: allow asg-web → asg-sql on 1433.
  • Augmented rules allow multiple prefixes/ports per rule, reducing rule sprawl.

Rule naming convention that helps audits:

<direction>-<allow|deny>-<source>-to-<destination>-<port/proto>-<ticket>
e.g. in-allow-asg-web-to-asg-sql-1433-tcp-CHG12345

Closing exposed management ports

Find them

// NSG rules allowing inbound management ports from the internet
resources
| where type =~ "microsoft.network/networksecuritygroups"
| mv-expand rule = properties.securityRules
| extend r = rule.properties
| where r.direction == "Inbound" and r.access == "Allow"
| extend src = tostring(r.sourceAddressPrefix), srcs = r.sourceAddressPrefixes,
         port = tostring(r.destinationPortRange), ports = r.destinationPortRanges
| where src in ("*", "0.0.0.0/0", "Internet", "Any") or set_has_element(srcs, "Internet") or set_has_element(srcs, "0.0.0.0/0")
| where port in ("22", "3389", "5985", "5986", "*") or set_has_element(ports, "22") or set_has_element(ports, "3389")
| project nsg = name, resourceGroup, rule = tostring(rule.name), src, port, priority = toint(r.priority)

Defender for Cloud also raises Management ports should be closed on your virtual machines and Internet-facing virtual machines should be protected with network security groups.

Replace access paths before removing rules

  1. Azure Bastion (Standard/Premium for native client and IP-based connections to on-prem/peered hosts) — browser or az network bastion ssh/rdp.
  2. Just-in-time VM access (Defender for Servers) — opens the port to a requester's IP for a limited time with approval and audit.
  3. VPN / ExpressRoute + jump hosts / PAWs for admins.
  4. Arc SSH access (az ssh arc) for Arc-enabled Linux and Windows servers — no inbound port needed on the server.

Then remove the rule, verify nothing broke (monitoring, backup, automation accounts often use the same port), and close the finding.


Private endpoints

A private endpoint puts a NIC with a private IP from your VNet in front of a PaaS resource. With public network access disabled, the resource is reachable only through that IP.

Terraform example: Key Vault

resource "azurerm_key_vault" "orders" {
  name                          = "kv-orders-prod"
  location                      = var.location
  resource_group_name           = var.rg
  tenant_id                     = var.tenant_id
  sku_name                      = "standard"
  enable_rbac_authorization     = true
  public_network_access_enabled = false
  purge_protection_enabled      = true

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
  }
}

resource "azurerm_private_endpoint" "kv_orders" {
  name                = "pe-kv-orders-prod"
  location            = var.location
  resource_group_name = var.rg
  subnet_id           = data.azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "psc-kv-orders-prod"
    private_connection_resource_id = azurerm_key_vault.orders.id
    subresource_names              = ["vault"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "default"
    private_dns_zone_ids = [data.azurerm_private_dns_zone.vault.id]   # privatelink.vaultcore.azure.net
  }
}

Sub-resource names and zones you'll use most

Service subresource_names Private DNS zone
Key Vault vault privatelink.vaultcore.azure.net
Storage (blob) blob privatelink.blob.core.windows.net
Storage (file) file privatelink.file.core.windows.net
Azure SQL sqlServer privatelink.database.windows.net
ACR registry privatelink.azurecr.io (plus regional data endpoints)
App Service (inbound) sites privatelink.azurewebsites.net
Service Bus / Event Hubs namespace privatelink.servicebus.windows.net
Azure Monitor (AMPLS) azuremonitor several (privatelink.monitor.azure.com, privatelink.oms.opinsights.azure.com, …)
Azure Arc (Private Link Scope) hybridcompute privatelink.his.arc.azure.com, privatelink.guestconfiguration.azure.com, privatelink.dp.kubernetesconfiguration.azure.com

Order of operations (avoid self-inflicted outages)

  1. Create the private endpoint and DNS records while public access is still enabled.
  2. Verify every consumer resolves the name to the private IP and connects successfully (including on-prem consumers, build agents, Functions, App Services via VNet integration).
  3. Check the resource's own logs for remaining public-IP clients.
  4. Then disable public network access.
  5. Keep a rollback: re-enabling public access is one setting — document it in the runbook.

NSGs on private endpoints

Network policies (NSG and UDR) for private endpoints are off by default on a subnet. To filter traffic to private endpoints with NSGs, enable them on the subnet:

az network vnet subnet update -g rg-net --vnet-name vnet-hub -n snet-private-endpoints \
  --private-endpoint-network-policies Enabled

Private DNS: where private endpoints go wrong

Almost every "private endpoint doesn't work" ticket is DNS.

How it should resolve:

kv-orders-prod.vault.azure.net
  → CNAME kv-orders-prod.privatelink.vaultcore.azure.net
      → (inside your network, private DNS zone)  A 10.20.30.4
      → (on the internet)                        public IP (blocked when public access disabled)

Patterns:

Consumer How it resolves privately
Azure VMs/App Services in linked VNets Private DNS zone linked to the VNet (Azure-provided DNS 168.63.129.16)
Hub-spoke with custom DNS servers Custom DNS forwards privatelink.* (or the public zones) to Azure DNS Private Resolver inbound endpoint or to 168.63.129.16 from a VM in a linked VNet
On-premises clients Conditional forwarders for the public zone names (e.g. vault.azure.net, blob.core.windows.net) → Private Resolver inbound endpoint

Forward the public zone (e.g. blob.core.windows.net), not the privatelink zone, from on-prem — clients query the public name and follow the CNAME.

Testing:

nslookup kv-orders-prod.vault.azure.net            # expect privatelink CNAME → 10.x
dig +short kv-orders-prod.vault.azure.net
Resolve-DnsName kv-orders-prod.vault.azure.net
Test-NetConnection kv-orders-prod.vault.azure.net -Port 443

Centralise private DNS zones in the hub (one zone per service type, linked to all spokes) and let Azure Policy create the privateDnsZoneGroup automatically when private endpoints are deployed — otherwise every team creates its own zone and resolution becomes inconsistent.


VNet and egress changes

Remediation sometimes needs bigger moves:

  • VNet integration for App Service/Functions so outbound calls reach private endpoints (az webapp vnet-integration add), with route-all enabled if egress must traverse the firewall.
  • Forced tunnelling / UDRs sending 0.0.0.0/0 to Azure Firewall — check that platform dependencies still work (AKS needs specific FQDNs and service tags; Windows activation needs KMS endpoints; Arc and Update Manager need their endpoints).
  • NAT gateway for predictable egress IPs (allow-listing at partners) and to remove default outbound access, which Azure is retiring for new VNets in favour of explicit outbound methods.
  • Subnet re-addressing / new subnets for private endpoints or delegated services — plan IP space with the network team early; this is often the long pole.

Network paths that patching and scanning depend on

When tightening egress, keep these working (verify current endpoint lists in Microsoft docs; they change):

Need Allow
Azure Arc agent Service tags AzureArcInfrastructure, AzureResourceManager, AzureActiveDirectory, GuestAndHybridManagement (or Arc Private Link Scope)
Azure Update Manager on Azure VMs Windows Update / Microsoft Update endpoints, or your WSUS; Linux distro repos or your mirror (*.ubuntu.com, RHUI for Azure RHEL images, etc.)
Defender for Endpoint / Defender for Servers MDE endpoints (streamlined connectivity list)
Azure Monitor Agent AzureMonitor service tag or AMPLS
Vulnerability scanners (Qualys/Tenable) Agent → cloud platform URLs; scanner appliance → targets on the scanned ports
Container builds / AKS ACR, MCR (mcr.microsoft.com, *.data.mcr.microsoft.com), package registries used in builds

A firewall change that quietly blocks the update path produces "compliant" servers that are simply no longer checking. Pair egress changes with the stale-assessment alert from Pipeline Scanning and Monitoring.


Writing a network change request the network team will love

Network teams receive vague requests ("open the firewall for the app") all day. A precise request gets done faster.

Title: Allow web tier → orders SQL on 1433 (replace broad 10.0.0.0/8 rule) — remediation VM-2026-00987

Driver:
  Finding VM-2026-00987 (P3): SQL port 1433 reachable from 10.0.0.0/8 on nsg-sql-prod.
  Target state: only web tier may reach SQL.

Change:
  ADD    nsg-sql-prod  in-allow-asg-web-to-asg-sql-1433-tcp-CHG12345
         priority 200, src ASG asg-web-prod, dst ASG asg-sql-prod, TCP 1433, Allow
  REMOVE nsg-sql-prod  in-allow-10.0.0.0-8-to-any-1433 (priority 300)

Who/what uses the path today (from VNet flow logs, last 30 days):
  10.20.1.0/24 (web VMs)        — expected, covered by new rule
  10.30.5.17 (backup server)    — uses 1433 for SQL backups → ADD rule for asg-backup
  10.40.2.9 (unknown)           — investigating, owner TBD; will not be covered

Validation:
  - Network Watcher IP flow verify: web → sql:1433 Allow; 10.40.2.9 → sql:1433 Deny
  - App smoke test: orders checkout end-to-end
  - Backup job succeeds next run

Rollback:
  Re-add previous rule (exported JSON attached), priority 300.

Window: <date/time>, standard change
Contacts: requester (remediation owner), app owner, DB owner

The single most valuable section is "who uses the path today" — pulled from flow logs. It turns a risky change into a known one.


Validating network changes

Before and after:

# Would this flow be allowed? (evaluates effective NSG rules on the NIC)
az network watcher test-ip-flow \
  --vm vm-web-01 -g rg-web --direction Outbound --protocol TCP \
  --local 10.20.1.4:50000 --remote 10.20.2.5:1433

# Effective security rules on a NIC (combined subnet + NIC NSGs)
az network nic list-effective-nsg -g rg-web -n vm-web-01-nic

# Effective routes (did a UDR send traffic to the firewall?)
az network nic show-effective-route-table -g rg-web -n vm-web-01-nic -o table

# End-to-end connectivity check from a VM (uses Network Watcher agent extension)
az network watcher test-connectivity -g rg-web --source-resource vm-web-01 \
  --dest-address kv-orders-prod.vault.azure.net --dest-port 443

From the hosts themselves:

nc -vz 10.20.2.5 1433
curl -sv https://kv-orders-prod.vault.azure.net/healthstatus -o /dev/null
Test-NetConnection 10.20.2.5 -Port 1433

Capture outputs in the change record — they're the closure evidence for the finding.


Flow logs and evidence

  • VNet flow logs replace NSG flow logs: NSG flow logs can no longer be created (since June 30, 2025) and are retired on September 30, 2027, when existing ones are deleted. Migrate now.
  • Enable traffic analytics on VNet flow logs for a queryable view (NTANetAnalytics table) of who talks to whom.
  • Use them before changes (who uses this path?) and after (did denied traffic appear that we didn't expect?).
NTANetAnalytics
| where TimeGenerated > ago(30d)
| where DestPort == 1433 and DestIp == "10.20.2.5"
| summarize flows = sum(AllowedInFlows + DeniedInFlows), lastSeen = max(TimeGenerated) by SrcIp
| order by flows desc

(Column names in traffic analytics tables can differ by schema version; run getschema to confirm.)


Runbooks

A runbook is a procedure written so that someone who didn't write it can execute it correctly, under pressure, and know when to stop.

Qualities of a good runbook

Quality What it looks like
Actionable Numbered steps with exact commands, not prose descriptions
Verifiable Every step that changes something has an expected result and a check
Safe Pre-checks, stop conditions, rollback section — before the steps start
Scoped Says what it covers and what it doesn't
Owned Named owner and last-reviewed date
Linked Alerts and tickets link to it; it links to the systems it touches
Tested Executed recently (game days, or literally the last patch cycle)

Types you need for this program

  • Monthly patching (per platform / per wave)
  • Emergency (P1/KEV) vulnerability remediation
  • Out-of-band patch deployment
  • Rollback procedures (snapshot restore, KB uninstall, package downgrade, slot swap, kubectl rollout undo)
  • Arc onboarding and troubleshooting
  • Making a PaaS service private (private endpoint migration)
  • Closing exposed management ports
  • Base image rebuild and redeploy
  • Exception request and review
  • Secret leak response (rotate → investigate → replace pattern)

Runbook template

---
title: "<Runbook name>"
date: YYYY-MM-DD
lastmod: YYYY-MM-DD
description: "<one line: what this does and when to use it>"
categories: ["vulnerability-management"]
tags: ["runbook", "<platform>", "<topic>"]
draft: false
---

# <Runbook name>

| | |
|---|---|
| **Purpose** | |
| **When to use** | Trigger(s): alert name, ticket type, schedule |
| **Scope** | Systems/environments covered — and explicitly not covered |
| **Owner** | Person/team |
| **Last reviewed / last executed** | YYYY-MM-DD / YYYY-MM-DD |
| **Estimated duration** | |
| **Risk** | Low / Medium / High — what could go wrong |
| **Change type** | Standard / Normal / Emergency |

## Prerequisites
- Access/roles required
- Tools (versions)
- Approvals

## Pre-checks
1. <check> — **Expected:** <result>. **If not:** stop / do X.

## Procedure
1. <step>
   ```bash
   <exact command>
   ```
   **Expected:** <output/result>
   **If not:** <what to do>

## Verification
- <how to prove success>

## Rollback
- **Trigger:** <conditions>
- **Steps:**
  1.

## Communication
- Before / during / after: who, which template

## Escalation
| Situation | Contact |
|---|---|

## References
- Related runbooks, docs, dashboards

## Change log
| Date | Change | By |
|---|---|---|

Example: monthly patching runbook

# Monthly Patching — Production Wave 2 (Azure + Arc)

| | |
|---|---|
| Purpose | Apply monthly security/critical updates to prod-wave2 servers |
| When to use | Patch Tuesday + 11 days, per change CHG-standard-PATCH-W2 |
| Scope | Servers tagged patch-group=prod-wave2 (Azure VMs and Arc). NOT clusters (see Wave 4 runbook) |
| Owner | Platform team |
| Duration | 4 hours window; ~30 min hands-on |
| Risk | Medium — reboots of non-redundant services listed in the pre-notice |

## Pre-checks (T-24h)
1. Ring 2 canaries patched with no open incidents. **Expected:** canary dashboard green.
2. Microsoft release health + distro errata reviewed for known issues. Record decisions in the change.
3. Pre-notice sent (template in Patch and Upgrade Management guide).
4. Resource Graph: confirm scope count matches expectations.
   ```kusto
   resources
   | where type in~ ("microsoft.compute/virtualmachines","microsoft.hybridcompute/machines")
   | where tags["patch-group"] == "prod-wave2"
   | summarize count() by type, tostring(properties.osType)
   ```
   **If count differs by > 5% from last month:** investigate before proceeding.

## Pre-checks (T-30m)
5. Pre-maintenance event handler healthy (last run succeeded). Snapshots enabled for tier-1.
6. Monitoring maintenance mode scheduled for the window.

## Procedure
7. Window starts automatically (maintenance configuration mc-prod-wave2). Post "[STARTED]" message.
8. At T+60m: check progress.
   ```kusto
   patchinstallationresources
   | where properties.lastModifiedDateTime > ago(2h)
   | summarize count() by tostring(properties.status)
   ```
   **Expected:** mostly InProgress/Succeeded. **If any Failed:** open the machine's history; triage (disk space, pending reboot, repo reachability).
9. At window end: post "[COMPLETED]" summary with counts and failures.

## Verification
10. Smoke tests from post-event: all passing.
11. Trigger on-demand assessment for failed/partial machines; schedule one-time updates in the catch-up window.
12. Dashboard: no machines in scope with pending security updates (except documented exceptions).

## Rollback
- Trigger: service health failing > 15 min after reboot with no quick fix, or app owner declares critical breakage.
- Steps: see Rollback runbook (snapshot restore for Azure VMs; KB/package rollback per platform).
- After rollback: raise exception for the unapplied update; post "[ISSUE]" update.

## Escalation
| Situation | Contact |
|---|---|
| > 5% install failures | Platform lead |
| Customer-facing impact | Incident manager (major incident process) |

Example: emergency vulnerability remediation runbook

# Emergency Remediation — Actively Exploited Vulnerability (P1 / KEV)

| | |
|---|---|
| When to use | CVE added to CISA KEV affecting our estate, vendor emergency advisory, or exploitation observed |
| Change type | Emergency |
| Target | Mitigated within hours; remediated within P1 SLA |

## 1. Scope (first 60 minutes)
1. Identify affected assets:
   - Servers: Defender for Cloud / MDVM software inventory search for product+version; Resource Graph
   - Images: registry/runtime findings for the CVE; `kubectl get pods -A -o jsonpath=...` for image refs
   - Network-only assets: Qualys/Tenable search by QID/plugin
2. Determine exposure per asset: internet-facing? reachable path? (NSG/Front Door/App Gateway config)
3. Open an incident/emergency change; page asset owners of exposed systems.

## 2. Mitigate (if a patch can't be deployed within hours)
4. Apply vendor mitigation (config change / feature disable) OR
5. Network block: NSG deny rule / WAF custom rule / disable public access. Record rule names for removal later.
6. Confirm mitigation with a test (vendor-provided check, scanner, or reproduction attempt in non-prod).

## 3. Remediate
7. Test patch on one non-prod system of each affected type.
8. Deploy: internet-facing first, then internal; use one-time updates / image rebuild + redeploy.
9. Reboot/restart as required; confirm running version.

## 4. Verify and hunt
10. On-demand assessment / re-scan; attach evidence to each finding.
11. Threat hunt for prior exploitation (EDR advanced hunting, web/app logs, sign-in logs) for the exposure period.
12. If indicators found → hand to incident response; preserve evidence.

## 5. Close
13. Remove temporary mitigations that are no longer needed (or convert to permanent controls via change).
14. Communicate completion to stakeholders.
15. Post-incident review within 5 business days: time to detect, scope, mitigate, remediate; what slowed us down.

Example: make a PaaS service private runbook

# Make a PaaS Resource Private (Private Endpoint Migration)

| | |
|---|---|
| When to use | Finding: "<service> should disable public network access" / "should use private link" |
| Scope | Key Vault, Storage, SQL, ACR, App Service (inbound), Service Bus/Event Hubs |
| Risk | High if consumers aren't inventoried — public access disable is the breaking step |

## Pre-checks
1. Inventory consumers: resource diagnostic logs (caller IPs) for 14–30 days; app config search; ask app owner.
   **Expected:** every caller identified as in-VNet, on-prem via ExpressRoute/VPN, or an Azure service with trusted-service bypass.
2. Confirm private DNS zone for the service exists in the hub and is linked to consumer VNets; on-prem forwarders configured.
3. Confirm subnet for private endpoints has free IPs.

## Procedure
4. Create private endpoint (Terraform PR) with DNS zone group. Apply.
5. From each consumer network: `nslookup <name>` → expect private IP. `Test-NetConnection <name> -Port 443` → success.
6. Monitor resource logs for 24–72 h: remaining public-IP callers?
   **If any:** identify and migrate them before continuing.
7. Disable public network access (Terraform PR: public_network_access_enabled = false).
8. Run consumer smoke tests.

## Verification
9. Defender recommendation for the resource turns healthy.
10. External test from internet: connection refused/forbidden.

## Rollback
- Re-enable public network access (single setting); leaves PE in place. Re-test consumers.

Keeping documentation alive

Documentation decays at the rate the systems change. Treat it like code:

  • Docs as code: Markdown in Git, merge requests for changes, review by someone who'll use it.
  • Front matter with lastmod and an explicit Last reviewed / last executed line; a CI job lists runbooks not reviewed in 6 months.
  • Update in the same change: if a change alters a procedure, the runbook update is part of the change's definition of done.
  • Link from alerts: every alert's description/annotation includes runbook_url. Unused runbooks are a signal; alerts with no runbook are a gap.
  • Execute to review: the best review is following the runbook literally during the next patch cycle or a game day, noting every place reality differed.
  • Retro feedback: after incidents and failed windows, update the runbook before closing the post-incident action items.
  • Link-check: run a link checker over the docs site in CI so references to moved docs and portals don't rot.

A runbook nobody has run in a year is a hypothesis. Schedule the tests.


References

  • Azure network security groups overview — Microsoft Learn
  • Azure Private Link and private endpoint DNS configuration — Microsoft Learn
  • Azure DNS Private Resolver — Microsoft Learn
  • Network Watcher (IP flow verify, effective security rules, connection troubleshoot) — Microsoft Learn
  • VNet flow logs and NSG flow logs retirement (September 30, 2027) — Microsoft Learn
  • Azure Bastion and Just-in-time VM access — Microsoft Learn
  • Google SRE Workbook, "On-Call" and "Incident Response" chapters (runbook practices)