Pipeline Scanning and Monitoring¶
Scanning in production tells you what's already exposed. Scanning in the pipeline stops new exposure from shipping. Monitoring connects the two so that a new finding or a regression (something you fixed coming back) turns into an alert for the right owner, quickly — instead of showing up in next month's report.
Contents¶
- Where scanning belongs
- Tool selection
- Gate policy: what fails a build
- Securing the scanners themselves
- GitLab CI example
- GitHub Actions example
- Azure Pipelines example
- SBOMs and signing
- Admission control: the last gate
- Defender for Cloud continuous export
- Alerting on new findings
- Detecting regressions
- Patch compliance and drift alerts
- Kubernetes in-cluster scanning with Prometheus
- Alert design
- Dashboards
Where scanning belongs¶
Developer Merge request Build Registry Deploy Runtime
───────── ───────────── ───── ──────── ────── ───────
pre-commit secrets (gitleaks) image scan Defender for admission Defender for
secrets SCA (deps) (trivy/grype) Containers policy Containers runtime,
IDE plugins IaC (checkov/ SBOM (syft) registry scan, (signed images, MDVM on servers,
trivy config) sign (notation/ re-scan on new allowed registry, trivy-operator,
SAST cosign) CVEs pod security) scheduled re-scans
Each stage catches what earlier ones can't:
- Merge request catches problems while the author still has context.
- Build sees the final image, including the base layers developers didn't write.
- Registry re-scans existing images as new CVEs are published — an image that was clean at build time won't stay clean.
- Admission stops anything that bypassed the pipeline.
- Runtime tells you what's actually exposed now.
Tool selection¶
| Purpose | Open source | Platform-native / commercial |
|---|---|---|
| Image vulnerabilities | Trivy, Grype | Defender for Containers (MDVM), Qualys Container Security, Tenable Cloud Security, GitLab Container Scanning |
| Dependencies (SCA) | Trivy fs, OSV-Scanner, npm audit, pip-audit, dotnet list package --vulnerable, cargo audit, govulncheck |
Dependabot, GitLab Dependency Scanning, Snyk, Mend |
| Secrets | gitleaks, trufflehog, Trivy --scanners secret |
GitHub secret scanning + push protection, GitLab Secret Detection |
| IaC (Terraform, Bicep, Kubernetes, Helm) | Checkov, Trivy config, kube-linter, tflint | Defender for Cloud DevOps security, GitLab IaC scanning |
| SAST | Semgrep, CodeQL | GitHub Advanced Security, GitLab SAST, SonarQube |
| SBOM | Syft, Trivy (--format cyclonedx/spdx-json) |
— |
| Signing | Notation (Notary Project), cosign (Sigstore) | ACR + Azure Key Vault Notation plugin |
Use one primary scanner per purpose in pipelines and keep it consistent with production. If CI uses Trivy and production uses Defender, expect small disagreements (different advisory sources, different handling of unfixed CVEs) and document which one is authoritative for gating.
Gate policy: what fails a build¶
Gates that fail too often get disabled. Gates that never fail aren't gates. Start narrow and tighten:
| Phase | Fail the build when… | Warn (don't fail) when… |
|---|---|---|
| Week 1 (adopt) | Secrets detected | Everything else — collect a baseline |
| Month 1 | Secrets; CRITICAL vulns with a fix available; any CISA KEV CVE | HIGH with fix; IaC misconfigs |
| Steady state | Secrets; CRITICAL + HIGH with fix; KEV; IaC: public exposure, missing encryption, privileged containers | MEDIUM with fix; unfixed CRITICAL/HIGH (route to exception process) |
Principles:
--ignore-unfixedfor gating. A developer can't fix what upstream hasn't fixed; unfixed findings go to the exception process and runtime monitoring.- New vs existing. On merge requests, fail only on findings introduced by the change (diff against the target branch's results) so one team isn't blocked by inherited debt — while scheduled pipelines report the full picture.
- Every suppression has an expiry and an exception ID (
.trivyignore.yamlwithexpired_atandstatement: "EXC-…"). - Break-glass exists, is logged, and is reviewed: a protected CI variable or label that downgrades the gate to warn, usable only by named roles.
Securing the scanners themselves¶
Security scanners run with access to source code, registry credentials and often cloud credentials — which makes them attractive targets.
In March 2026, attackers who had stolen credentials in an earlier incident force-pushed malicious commits to most version tags of aquasecurity/trivy-action and all tags of aquasecurity/setup-trivy, published a malicious Trivy release, and later pushed malicious Trivy images to Docker Hub. Pipelines that referenced the action by tag ran credential-stealing code. The incident is tracked as CVE-2026-33634 (GHSA-69fq-xp46-6x23) and was added to CISA KEV. A parallel compromise hit Aqua's kics-github-action.
Lessons that apply to every tool in this guide:
- Pin third-party actions/components by full commit SHA, not tag. Tags are mutable.
yaml - uses: aquasecurity/trivy-action@<40-char-commit-sha> # vX.Y.Z — verified YYYY-MM-DDLet Dependabot/Renovate update the SHA via reviewed PRs. - Pin container images by digest (
aquasec/trivy:<tag>@sha256:…) and verify release signatures/checksums for downloaded binaries. - Least-privilege tokens:
permissions: contents: readby default in GitHub Actions; scoped, short-lived tokens in GitLab; OIDC federation to Azure instead of stored client secrets. - Separate scanning from deploying. The job that runs a third-party scanner shouldn't hold production deployment credentials.
- Avoid
pull_request_target(GitHub) with checkout of untrusted code — that's how the original credential theft happened. - Consider mirroring critical tools into your own registry/artifact store after verification, and running from there.
- Audit: if you used the affected Trivy components in the incident window, follow the advisory — check run logs and rotate every secret those pipelines could reach.
GitLab CI example¶
A self-contained pipeline using pinned images. Replace placeholder digests with verified ones.
# .gitlab-ci.yml
stages: [secrets, test, build, scan, sign, deploy]
variables:
IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
TRIVY_IMAGE: "aquasec/trivy:<version>@sha256:<verified-digest>"
GITLEAKS_IMAGE: "zricethezav/gitleaks:<version>@sha256:<verified-digest>"
SYFT_IMAGE: "anchore/syft:<version>@sha256:<verified-digest>"
TRIVY_NO_PROGRESS: "true"
TRIVY_CACHE_DIR: ".trivycache/"
secrets:
stage: secrets
image: { name: "$GITLEAKS_IMAGE", entrypoint: [""] }
script:
- gitleaks git --redact --report-format sarif --report-path gitleaks.sarif .
artifacts:
when: always
paths: [gitleaks.sarif]
deps-and-iac:
stage: test
image: { name: "$TRIVY_IMAGE", entrypoint: [""] }
script:
# Dependencies (lock files) — gate on fixable HIGH/CRITICAL
- trivy fs --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 .
# IaC misconfigurations — report only while adopting
- trivy config --severity HIGH,CRITICAL --exit-code 0 --format table ./infra
cache:
key: trivy
paths: [.trivycache/]
build:
stage: build
image: docker:27
services: [docker:27-dind]
script:
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
- docker build --pull -t "$IMAGE" .
- docker push "$IMAGE"
- docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE" > image-digest.txt
artifacts:
paths: [image-digest.txt]
image-scan:
stage: scan
image: { name: "$TRIVY_IMAGE", entrypoint: [""] }
variables:
TRIVY_USERNAME: "$CI_REGISTRY_USER"
TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
script:
- REF=$(cat image-digest.txt)
# Full report as artifact (all severities, incl. unfixed)
- trivy image --format json --output trivy-image.json "$REF"
# GitLab security dashboard format
- trivy image --format template --template "@/contrib/gitlab.tpl" --output gl-container-scanning-report.json "$REF"
# Gate: fixable HIGH/CRITICAL and secrets in layers
- trivy image --scanners vuln,secret --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$REF"
artifacts:
when: always
paths: [trivy-image.json]
reports:
container_scanning: gl-container-scanning-report.json
sbom:
stage: scan
image: { name: "$SYFT_IMAGE", entrypoint: [""] }
script:
- syft "$(cat image-digest.txt)" -o cyclonedx-json=sbom.cdx.json
artifacts:
paths: [sbom.cdx.json]
nightly-rescan:
stage: scan
image: { name: "$TRIVY_IMAGE", entrypoint: [""] }
rules:
- if: $CI_PIPELINE_SOURCE == "schedule" && $RESCAN == "true"
script:
# Re-scan what's deployed (digest list maintained by the deploy job / GitOps repo)
- for ref in $(cat deployed-images.txt); do trivy image --severity HIGH,CRITICAL --ignore-unfixed --format json --output "rescan-$(echo $ref | sha256sum | cut -c1-12).json" "$ref"; done
artifacts:
paths: ["rescan-*.json"]
Notes:
- The
contrib/gitlab.tpltemplate ships inside the Trivy image; on GitLab Ultimate you can alternatively include the managedJobs/Container-Scanning.gitlab-ci.ymltemplate. - Put shared scanning jobs in a central CI template project and
include:them from every repo, so policy changes happen in one place (and every repo's header stays in sync).
.trivyignore.yaml (committed, reviewed via CODEOWNERS):
vulnerabilities:
- id: CVE-2026-XXXXX
statement: "EXC-2026-0057 — not reachable; upstream fix pending"
expired_at: 2026-12-31
GitHub Actions example¶
# .github/workflows/build-scan.yml
name: build-scan
on:
pull_request:
push:
branches: [main]
schedule:
- cron: "17 6 * * *" # daily re-scan
permissions:
contents: read
jobs:
build-and-scan:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
security-events: write # SARIF upload
id-token: write # OIDC to Azure, if pushing to ACR
steps:
- uses: actions/checkout@<sha> # vX — pinned
with: { persist-credentials: false }
- name: Secrets scan
uses: gitleaks/gitleaks-action@<sha> # vX — pinned
env: { GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}" }
- name: Build
run: docker build --pull -t app:${{ github.sha }} .
- name: Install Trivy (pinned, checksum-verified)
run: |
VER="<version>"
curl -fsSLO "https://github.com/aquasecurity/trivy/releases/download/v${VER}/trivy_${VER}_Linux-64bit.tar.gz"
curl -fsSLO "https://github.com/aquasecurity/trivy/releases/download/v${VER}/trivy_${VER}_checksums.txt"
grep "trivy_${VER}_Linux-64bit.tar.gz" "trivy_${VER}_checksums.txt" | sha256sum -c -
tar -xzf "trivy_${VER}_Linux-64bit.tar.gz" trivy && sudo mv trivy /usr/local/bin/
- name: Scan image (SARIF for code scanning)
run: trivy image --format sarif --output trivy.sarif app:${{ github.sha }}
- uses: github/codeql-action/upload-sarif@<sha> # pinned
if: always()
with: { sarif_file: trivy.sarif }
- name: Gate
run: trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 app:${{ github.sha }}
Installing the binary directly with checksum verification avoids depending on a third-party action at all — a reasonable choice after the 2026 incident. For stronger assurance, also verify the release's Sigstore signature with cosign verify-blob.
Azure Pipelines example¶
# azure-pipelines.yml (excerpt)
trigger: [main]
schedules:
- cron: "0 11 * * *"
displayName: Daily re-scan
branches: { include: [main] }
always: true
pool: { vmImage: ubuntu-latest }
variables:
acrName: acrprod
imageRepo: orders-api
tag: $(Build.BuildId)
trivyVersion: "<version>"
steps:
- task: AzureCLI@2
displayName: Build and push to ACR
inputs:
azureSubscription: sc-acr-push # workload identity federation service connection
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az acr login -n $(acrName)
docker build --pull -t $(acrName).azurecr.io/$(imageRepo):$(tag) .
docker push $(acrName).azurecr.io/$(imageRepo):$(tag)
- bash: |
set -euo pipefail
curl -fsSLO "https://github.com/aquasecurity/trivy/releases/download/v$(trivyVersion)/trivy_$(trivyVersion)_Linux-64bit.tar.gz"
curl -fsSLO "https://github.com/aquasecurity/trivy/releases/download/v$(trivyVersion)/trivy_$(trivyVersion)_checksums.txt"
grep "trivy_$(trivyVersion)_Linux-64bit.tar.gz" "trivy_$(trivyVersion)_checksums.txt" | sha256sum -c -
tar -xzf trivy_*_Linux-64bit.tar.gz trivy
./trivy image --format json --output $(Build.ArtifactStagingDirectory)/trivy.json \
$(acrName).azurecr.io/$(imageRepo):$(tag)
./trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 \
$(acrName).azurecr.io/$(imageRepo):$(tag)
displayName: Scan image
- publish: $(Build.ArtifactStagingDirectory)
artifact: scan-results
condition: always()
Defender for Cloud's DevOps security connectors (Azure DevOps, GitHub, GitLab) can also pull pipeline and repository findings into Defender for Cloud, giving a single view from code to runtime.
SBOMs and signing¶
An SBOM turns "are we affected by CVE-X?" from a scramble into a query.
- Generate one per image at build (
syftortrivy --format cyclonedx). - Store it next to the image as an OCI artifact (ORAS attach /
notation/cosign attest) and in a searchable place (Dependency-Track, GUAC, or simply object storage +grype sbom:re-scans). - Re-scan SBOMs daily against updated vulnerability databases:
grype sbom:./sbom.cdx.json --fail-on highis fast and doesn't need the image.
Sign images after they pass the gate, so the signature means "built by our pipeline and passed policy":
# Notation with Azure Key Vault plugin (ACR)
notation sign --signature-format cose \
--id "https://kv-signing.vault.azure.net/keys/image-signing/<version>" \
--plugin azure-kv \
acrprod.azurecr.io/orders-api@sha256:<digest>
Admission control: the last gate¶
Pipelines can be bypassed (manual kubectl apply, a forgotten repo). Enforce at the cluster:
- Allowed registries: Azure Policy for AKS — only
acrprod.azurecr.io/*. - Signed images only: AKS image integrity (Ratify + Azure Policy) verifying Notation signatures.
- Pod security: Pod Security Admission
restricted, or the Azure Policy initiative. - Defender for Containers gated deployment (where available): block or audit deployments of images with vulnerability findings above a threshold.
Start every admission policy in audit mode, review what it would have blocked, then switch to deny.
Defender for Cloud continuous export¶
Defender for Cloud's portal shows current state; continuous export gives you history and event streams for alerting. Export:
- Security recommendations (assessments) and security findings (sub-assessments — the individual CVEs) → Log Analytics (
SecurityRecommendation,SecurityNestedRecommendationtables) and/or Event Hubs. - Security alerts → your SIEM.
- Secure score and regulatory compliance snapshots → trend reporting.
Terraform:
resource "azurerm_security_center_automation" "export_findings" {
name = "export-findings-to-law"
location = azurerm_resource_group.security.location
resource_group_name = azurerm_resource_group.security.name
scopes = ["/subscriptions/${var.subscription_id}"]
action {
type = "LogAnalytics"
resource_id = azurerm_log_analytics_workspace.security.id
}
source {
event_source = "Assessments"
}
source {
event_source = "SubAssessments"
}
source {
event_source = "AssessmentsSnapshot"
}
source {
event_source = "SubAssessmentsSnapshot"
}
}
Snapshots give you a weekly full state; the non-snapshot sources stream changes as they happen. Deploy the export per subscription (or via the built-in Deploy export to Log Analytics workspace for Microsoft Defender for Cloud data policy at management-group scope).
Alerting on new findings¶
Column names in exported tables have shifted over time; check the schema in your workspace (SecurityNestedRecommendation | getschema) and adjust.
New critical/high server or container findings in the last hour:
SecurityNestedRecommendation
| where TimeGenerated > ago(1h)
| extend sev = tostring(coalesce(RecommendationSeverity, parse_json(AdditionalData).severity))
| where RecommendationState == "Unhealthy" and sev in ("High", "Critical")
| summarize firstSeen = min(TimeGenerated), resources = dcount(AssessedResourceId)
by Finding = RecommendationName, sev
| order by sev asc, resources desc
KEV join — keep a small KEV_CL custom table refreshed daily (Logic App or Function pulling the CISA JSON), then:
SecurityNestedRecommendation
| where TimeGenerated > ago(1d) and RecommendationState == "Unhealthy"
| extend cve = extract(@"(CVE-\d{4}-\d{4,})", 1, strcat(RecommendationName, " ", tostring(AdditionalData)))
| where isnotempty(cve)
| join kind=inner (KEV_CL | project cve = cveID_s, dueDate = dueDate_s) on cve
| summarize resources = make_set(AssessedResourceId, 50) by cve, dueDate
Create these as Azure Monitor scheduled log search alerts with an action group that routes by owner (see Alert design). For "something changed" signals without polling, Defender for Cloud workflow automation can trigger a Logic App directly when a recommendation or alert fires — useful for creating tickets automatically.
You can also query Azure Resource Graph from a log search alert using arg("").securityresources, avoiding export latency for current-state checks.
Detecting regressions¶
A regression is a finding that was closed and came back. Common causes:
- Autoscaling or redeploys from an old image or golden image.
- Package downgraded or pinned by a config change.
- A configuration baseline reverted by a vendor installer or manual change.
- A new deployment of an old app version (rollback without re-fixing).
Detection approaches:
- Tracker state machine — your normalised finding store marks
Closed – Remediated→Reopenedwhen the same(asset, CVE, component)reappears. Alert on every reopen for P1/P2. - Scanner-native — Tenable and Qualys both expose Reopened/Re-Opened states; include them in exports and alerts.
- Log Analytics comparison — compare the latest snapshot with the previous week's:
let thisWeek = SecurityNestedRecommendation
| where TimeGenerated > ago(7d) and RecommendationState == "Unhealthy"
| distinct AssessedResourceId, RecommendationName;
let lastWeekHealthy = SecurityNestedRecommendation
| where TimeGenerated between (ago(14d) .. ago(7d)) and RecommendationState == "Healthy"
| distinct AssessedResourceId, RecommendationName;
thisWeek
| join kind=inner lastWeekHealthy on AssessedResourceId, RecommendationName
| project AssessedResourceId, RecommendationName
- Image digest drift — alert when a running pod's image digest isn't one produced by the pipeline in the last N days (indicates an old image resurfacing).
- Golden image age — alert when the newest image version in the Compute Gallery is older than ~35 days.
Patch compliance and drift alerts¶
Azure Update Manager — log alert using Resource Graph:
arg("").patchassessmentresources
| where type endswith "/patchassessmentresults"
| extend critical = toint(properties.availablePatchCountByClassification.critical),
security = toint(properties.availablePatchCountByClassification.security),
lastAssessed = todatetime(properties.lastModifiedDateTime)
| where (critical + security) > 0 and lastAssessed > ago(2d)
| extend machine = tostring(split(id, "/patchAssessmentResults/")[0])
| project machine, critical, security
Suggested alerts:
| Alert | Condition | Severity |
|---|---|---|
| Patch run failed | Any patchinstallationresults with status Failed in last 24 h |
Sev 2 |
| Stale assessment | lastModifiedDateTime older than 3 days |
Sev 3 |
| Reboot pending too long | rebootPending == true for > 7 days |
Sev 3 |
| Security updates outstanding past SLA | Pending security updates on a machine for > 30 days (track first-seen in a custom table) | Sev 2 |
| Arc agent disconnected | microsoft.hybridcompute/machines with status != Connected for > 24 h |
Sev 3 |
Configuration drift — Azure Policy compliance state changes are published to Event Grid (Microsoft.PolicyInsights.PolicyStateChanged). Subscribe a Function/Logic App filtered to your baseline assignments to alert when a machine goes non-compliant, rather than waiting for a report. For high-signal changes (admin group membership, security agent stopped), alert from Change Tracking / SIEM as described in Configuration Management.
Kubernetes in-cluster scanning with Prometheus¶
Outside Azure — or alongside Defender in a homelab or non-AKS cluster — Trivy Operator continuously scans workloads and exposes results as CRDs and Prometheus metrics.
helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm upgrade --install trivy-operator aqua/trivy-operator \
--namespace trivy-system --create-namespace \
--version <pinned-chart-version> \
--set serviceMonitor.enabled=true \
--set trivy.ignoreUnfixed=true
(Pin and verify chart and image versions — the same supply-chain lessons apply.)
kubectl get vulnerabilityreports -A -o wide
kubectl get configauditreports -A -o wide
kubectl get exposedsecretreports -A
Alerting rules:
groups:
- name: vulnerability-management
rules:
- alert: CriticalVulnerabilitiesInRunningImage
expr: sum by (namespace, image_repository, image_tag) (trivy_image_vulnerabilities{severity="Critical"}) > 0
for: 1h
labels: { severity: warning }
annotations:
summary: "Critical CVEs in {{ $labels.namespace }}/{{ $labels.image_repository }}:{{ $labels.image_tag }}"
runbook_url: "https://docs.example.com/docs/vulnerability-management/container-and-application-remediation/"
- alert: ExposedSecretInImage
expr: sum by (namespace, image_repository) (trivy_image_exposedsecrets) > 0
for: 15m
labels: { severity: critical }
annotations:
summary: "Secret found in image {{ $labels.image_repository }} ({{ $labels.namespace }})"
For hosts, the node_exporter textfile collector scripts (apt_info, yum) expose pending updates and reboot-required metrics; see the Linux unattended upgrades guide for details.
Alert design¶
Alerts that go to a shared mailbox get ignored. Design for the owner.
| Principle | How |
|---|---|
| Route to owners | Resolve the resource's owner tag (or CODEOWNERS for images) and route via action groups / Alertmanager receivers per team |
| Aggregate | One alert per (finding, team) per day, listing affected resources — not one per resource |
| Severity follows priority | P1/KEV → page; P2 → ticket + channel; P3/P4 → weekly digest |
| Every alert has a runbook link | runbook_url annotation / alert description linking to the procedure |
| SLA-aware | Alert at 75% of SLA elapsed, not just at creation |
| Self-resolving | Alerts close when the condition clears (verification) |
| Measure noise | Track alerts per team per week and the % acted on; prune or tune alerts nobody acts on |
Dashboards¶
One dashboard per audience:
Engineering teams (daily) - My open findings by priority and due date - Findings breaching SLA in the next 7 days - My images with fixable HIGH/CRITICAL - Reopened findings
Platform / remediation owner (daily/weekly) - New findings (24 h, 7 d) by source - KEV exposure count and age - Patch compliance by ring and environment - Failed patch runs, stale assessments, disconnected agents - Pipeline gate failures and break-glass uses
Leadership (monthly) - SLA adherence trend by priority - MTTR trend - Open KEV count - Exceptions: count, age, expired - Coverage: % assets scanned
Azure Workbooks over Log Analytics + Resource Graph cover the Azure side; Grafana works well if you're also pulling Prometheus metrics and scanner exports into the same place.
References¶
- Trivy documentation and GitHub advisory GHSA-69fq-xp46-6x23 (March 2026 compromise)
- CVE-2026-33634 — NVD
- GitHub Actions security hardening: pinning actions to a full-length commit SHA — docs.github.com
- Defender for Cloud continuous export — Microsoft Learn
- Defender for Cloud DevOps security — Microsoft Learn
- Azure Monitor log search alerts with Azure Resource Graph — Microsoft Learn
- Trivy Operator — aquasecurity.github.io/trivy-operator
- Notary Project / Notation and AKS image integrity — Microsoft Learn