Andrew Mercer
on this page

Pipeline Scanning and Monitoring

Scanning in production tells you what's already exposed. Scanning in the pipeline stops new exposure from shipping. Monitoring connects the two so that a new finding or a regression (something you fixed coming back) turns into an alert for the right owner, quickly — instead of showing up in next month's report.


Contents

  1. Where scanning belongs
  2. Tool selection
  3. Gate policy: what fails a build
  4. Securing the scanners themselves
  5. GitLab CI example
  6. GitHub Actions example
  7. Azure Pipelines example
  8. SBOMs and signing
  9. Admission control: the last gate
  10. Defender for Cloud continuous export
  11. Alerting on new findings
  12. Detecting regressions
  13. Patch compliance and drift alerts
  14. Kubernetes in-cluster scanning with Prometheus
  15. Alert design
  16. Dashboards

Where scanning belongs

 Developer        Merge request         Build               Registry            Deploy             Runtime
 ─────────        ─────────────         ─────               ────────            ──────             ───────
 pre-commit       secrets (gitleaks)    image scan          Defender for        admission          Defender for
 secrets          SCA (deps)            (trivy/grype)       Containers          policy             Containers runtime,
 IDE plugins      IaC (checkov/         SBOM (syft)         registry scan,      (signed images,    MDVM on servers,
                  trivy config)         sign (notation/     re-scan on new      allowed registry,  trivy-operator,
                  SAST                  cosign)             CVEs                pod security)      scheduled re-scans

Each stage catches what earlier ones can't:

  • Merge request catches problems while the author still has context.
  • Build sees the final image, including the base layers developers didn't write.
  • Registry re-scans existing images as new CVEs are published — an image that was clean at build time won't stay clean.
  • Admission stops anything that bypassed the pipeline.
  • Runtime tells you what's actually exposed now.

Tool selection

Purpose Open source Platform-native / commercial
Image vulnerabilities Trivy, Grype Defender for Containers (MDVM), Qualys Container Security, Tenable Cloud Security, GitLab Container Scanning
Dependencies (SCA) Trivy fs, OSV-Scanner, npm audit, pip-audit, dotnet list package --vulnerable, cargo audit, govulncheck Dependabot, GitLab Dependency Scanning, Snyk, Mend
Secrets gitleaks, trufflehog, Trivy --scanners secret GitHub secret scanning + push protection, GitLab Secret Detection
IaC (Terraform, Bicep, Kubernetes, Helm) Checkov, Trivy config, kube-linter, tflint Defender for Cloud DevOps security, GitLab IaC scanning
SAST Semgrep, CodeQL GitHub Advanced Security, GitLab SAST, SonarQube
SBOM Syft, Trivy (--format cyclonedx/spdx-json) —
Signing Notation (Notary Project), cosign (Sigstore) ACR + Azure Key Vault Notation plugin

Use one primary scanner per purpose in pipelines and keep it consistent with production. If CI uses Trivy and production uses Defender, expect small disagreements (different advisory sources, different handling of unfixed CVEs) and document which one is authoritative for gating.


Gate policy: what fails a build

Gates that fail too often get disabled. Gates that never fail aren't gates. Start narrow and tighten:

Phase Fail the build when… Warn (don't fail) when…
Week 1 (adopt) Secrets detected Everything else — collect a baseline
Month 1 Secrets; CRITICAL vulns with a fix available; any CISA KEV CVE HIGH with fix; IaC misconfigs
Steady state Secrets; CRITICAL + HIGH with fix; KEV; IaC: public exposure, missing encryption, privileged containers MEDIUM with fix; unfixed CRITICAL/HIGH (route to exception process)

Principles:

  • --ignore-unfixed for gating. A developer can't fix what upstream hasn't fixed; unfixed findings go to the exception process and runtime monitoring.
  • New vs existing. On merge requests, fail only on findings introduced by the change (diff against the target branch's results) so one team isn't blocked by inherited debt — while scheduled pipelines report the full picture.
  • Every suppression has an expiry and an exception ID (.trivyignore.yaml with expired_at and statement: "EXC-…").
  • Break-glass exists, is logged, and is reviewed: a protected CI variable or label that downgrades the gate to warn, usable only by named roles.

Securing the scanners themselves

Security scanners run with access to source code, registry credentials and often cloud credentials — which makes them attractive targets.

In March 2026, attackers who had stolen credentials in an earlier incident force-pushed malicious commits to most version tags of aquasecurity/trivy-action and all tags of aquasecurity/setup-trivy, published a malicious Trivy release, and later pushed malicious Trivy images to Docker Hub. Pipelines that referenced the action by tag ran credential-stealing code. The incident is tracked as CVE-2026-33634 (GHSA-69fq-xp46-6x23) and was added to CISA KEV. A parallel compromise hit Aqua's kics-github-action.

Lessons that apply to every tool in this guide:

  1. Pin third-party actions/components by full commit SHA, not tag. Tags are mutable. yaml - uses: aquasecurity/trivy-action@<40-char-commit-sha> # vX.Y.Z — verified YYYY-MM-DD Let Dependabot/Renovate update the SHA via reviewed PRs.
  2. Pin container images by digest (aquasec/trivy:<tag>@sha256:…) and verify release signatures/checksums for downloaded binaries.
  3. Least-privilege tokens: permissions: contents: read by default in GitHub Actions; scoped, short-lived tokens in GitLab; OIDC federation to Azure instead of stored client secrets.
  4. Separate scanning from deploying. The job that runs a third-party scanner shouldn't hold production deployment credentials.
  5. Avoid pull_request_target (GitHub) with checkout of untrusted code — that's how the original credential theft happened.
  6. Consider mirroring critical tools into your own registry/artifact store after verification, and running from there.
  7. Audit: if you used the affected Trivy components in the incident window, follow the advisory — check run logs and rotate every secret those pipelines could reach.

GitLab CI example

A self-contained pipeline using pinned images. Replace placeholder digests with verified ones.

# .gitlab-ci.yml
stages: [secrets, test, build, scan, sign, deploy]

variables:
  IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
  TRIVY_IMAGE: "aquasec/trivy:<version>@sha256:<verified-digest>"
  GITLEAKS_IMAGE: "zricethezav/gitleaks:<version>@sha256:<verified-digest>"
  SYFT_IMAGE: "anchore/syft:<version>@sha256:<verified-digest>"
  TRIVY_NO_PROGRESS: "true"
  TRIVY_CACHE_DIR: ".trivycache/"

secrets:
  stage: secrets
  image: { name: "$GITLEAKS_IMAGE", entrypoint: [""] }
  script:
    - gitleaks git --redact --report-format sarif --report-path gitleaks.sarif .
  artifacts:
    when: always
    paths: [gitleaks.sarif]

deps-and-iac:
  stage: test
  image: { name: "$TRIVY_IMAGE", entrypoint: [""] }
  script:
    # Dependencies (lock files) — gate on fixable HIGH/CRITICAL
    - trivy fs --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 .
    # IaC misconfigurations — report only while adopting
    - trivy config --severity HIGH,CRITICAL --exit-code 0 --format table ./infra
  cache:
    key: trivy
    paths: [.trivycache/]

build:
  stage: build
  image: docker:27
  services: [docker:27-dind]
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - docker build --pull -t "$IMAGE" .
    - docker push "$IMAGE"
    - docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE" > image-digest.txt
  artifacts:
    paths: [image-digest.txt]

image-scan:
  stage: scan
  image: { name: "$TRIVY_IMAGE", entrypoint: [""] }
  variables:
    TRIVY_USERNAME: "$CI_REGISTRY_USER"
    TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
  script:
    - REF=$(cat image-digest.txt)
    # Full report as artifact (all severities, incl. unfixed)
    - trivy image --format json --output trivy-image.json "$REF"
    # GitLab security dashboard format
    - trivy image --format template --template "@/contrib/gitlab.tpl" --output gl-container-scanning-report.json "$REF"
    # Gate: fixable HIGH/CRITICAL and secrets in layers
    - trivy image --scanners vuln,secret --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$REF"
  artifacts:
    when: always
    paths: [trivy-image.json]
    reports:
      container_scanning: gl-container-scanning-report.json

sbom:
  stage: scan
  image: { name: "$SYFT_IMAGE", entrypoint: [""] }
  script:
    - syft "$(cat image-digest.txt)" -o cyclonedx-json=sbom.cdx.json
  artifacts:
    paths: [sbom.cdx.json]

nightly-rescan:
  stage: scan
  image: { name: "$TRIVY_IMAGE", entrypoint: [""] }
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" && $RESCAN == "true"
  script:
    # Re-scan what's deployed (digest list maintained by the deploy job / GitOps repo)
    - for ref in $(cat deployed-images.txt); do trivy image --severity HIGH,CRITICAL --ignore-unfixed --format json --output "rescan-$(echo $ref | sha256sum | cut -c1-12).json" "$ref"; done
  artifacts:
    paths: ["rescan-*.json"]

Notes:

  • The contrib/gitlab.tpl template ships inside the Trivy image; on GitLab Ultimate you can alternatively include the managed Jobs/Container-Scanning.gitlab-ci.yml template.
  • Put shared scanning jobs in a central CI template project and include: them from every repo, so policy changes happen in one place (and every repo's header stays in sync).

.trivyignore.yaml (committed, reviewed via CODEOWNERS):

vulnerabilities:
  - id: CVE-2026-XXXXX
    statement: "EXC-2026-0057 — not reachable; upstream fix pending"
    expired_at: 2026-12-31

GitHub Actions example

# .github/workflows/build-scan.yml
name: build-scan
on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "17 6 * * *"   # daily re-scan

permissions:
  contents: read

jobs:
  build-and-scan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      security-events: write   # SARIF upload
      id-token: write          # OIDC to Azure, if pushing to ACR
    steps:
      - uses: actions/checkout@<sha>            # vX — pinned
        with: { persist-credentials: false }

      - name: Secrets scan
        uses: gitleaks/gitleaks-action@<sha>    # vX — pinned
        env: { GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}" }

      - name: Build
        run: docker build --pull -t app:${{ github.sha }} .

      - name: Install Trivy (pinned, checksum-verified)
        run: |
          VER="<version>"
          curl -fsSLO "https://github.com/aquasecurity/trivy/releases/download/v${VER}/trivy_${VER}_Linux-64bit.tar.gz"
          curl -fsSLO "https://github.com/aquasecurity/trivy/releases/download/v${VER}/trivy_${VER}_checksums.txt"
          grep "trivy_${VER}_Linux-64bit.tar.gz" "trivy_${VER}_checksums.txt" | sha256sum -c -
          tar -xzf "trivy_${VER}_Linux-64bit.tar.gz" trivy && sudo mv trivy /usr/local/bin/

      - name: Scan image (SARIF for code scanning)
        run: trivy image --format sarif --output trivy.sarif app:${{ github.sha }}

      - uses: github/codeql-action/upload-sarif@<sha>   # pinned
        if: always()
        with: { sarif_file: trivy.sarif }

      - name: Gate
        run: trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 app:${{ github.sha }}

Installing the binary directly with checksum verification avoids depending on a third-party action at all — a reasonable choice after the 2026 incident. For stronger assurance, also verify the release's Sigstore signature with cosign verify-blob.


Azure Pipelines example

# azure-pipelines.yml (excerpt)
trigger: [main]
schedules:
  - cron: "0 11 * * *"
    displayName: Daily re-scan
    branches: { include: [main] }
    always: true

pool: { vmImage: ubuntu-latest }

variables:
  acrName: acrprod
  imageRepo: orders-api
  tag: $(Build.BuildId)
  trivyVersion: "<version>"

steps:
  - task: AzureCLI@2
    displayName: Build and push to ACR
    inputs:
      azureSubscription: sc-acr-push       # workload identity federation service connection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        az acr login -n $(acrName)
        docker build --pull -t $(acrName).azurecr.io/$(imageRepo):$(tag) .
        docker push $(acrName).azurecr.io/$(imageRepo):$(tag)

  - bash: |
      set -euo pipefail
      curl -fsSLO "https://github.com/aquasecurity/trivy/releases/download/v$(trivyVersion)/trivy_$(trivyVersion)_Linux-64bit.tar.gz"
      curl -fsSLO "https://github.com/aquasecurity/trivy/releases/download/v$(trivyVersion)/trivy_$(trivyVersion)_checksums.txt"
      grep "trivy_$(trivyVersion)_Linux-64bit.tar.gz" "trivy_$(trivyVersion)_checksums.txt" | sha256sum -c -
      tar -xzf trivy_*_Linux-64bit.tar.gz trivy
      ./trivy image --format json --output $(Build.ArtifactStagingDirectory)/trivy.json \
        $(acrName).azurecr.io/$(imageRepo):$(tag)
      ./trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 \
        $(acrName).azurecr.io/$(imageRepo):$(tag)
    displayName: Scan image

  - publish: $(Build.ArtifactStagingDirectory)
    artifact: scan-results
    condition: always()

Defender for Cloud's DevOps security connectors (Azure DevOps, GitHub, GitLab) can also pull pipeline and repository findings into Defender for Cloud, giving a single view from code to runtime.


SBOMs and signing

An SBOM turns "are we affected by CVE-X?" from a scramble into a query.

  • Generate one per image at build (syft or trivy --format cyclonedx).
  • Store it next to the image as an OCI artifact (ORAS attach / notation/cosign attest) and in a searchable place (Dependency-Track, GUAC, or simply object storage + grype sbom: re-scans).
  • Re-scan SBOMs daily against updated vulnerability databases: grype sbom:./sbom.cdx.json --fail-on high is fast and doesn't need the image.

Sign images after they pass the gate, so the signature means "built by our pipeline and passed policy":

# Notation with Azure Key Vault plugin (ACR)
notation sign --signature-format cose \
  --id "https://kv-signing.vault.azure.net/keys/image-signing/<version>" \
  --plugin azure-kv \
  acrprod.azurecr.io/orders-api@sha256:<digest>

Admission control: the last gate

Pipelines can be bypassed (manual kubectl apply, a forgotten repo). Enforce at the cluster:

  • Allowed registries: Azure Policy for AKS — only acrprod.azurecr.io/*.
  • Signed images only: AKS image integrity (Ratify + Azure Policy) verifying Notation signatures.
  • Pod security: Pod Security Admission restricted, or the Azure Policy initiative.
  • Defender for Containers gated deployment (where available): block or audit deployments of images with vulnerability findings above a threshold.

Start every admission policy in audit mode, review what it would have blocked, then switch to deny.


Defender for Cloud continuous export

Defender for Cloud's portal shows current state; continuous export gives you history and event streams for alerting. Export:

  • Security recommendations (assessments) and security findings (sub-assessments — the individual CVEs) → Log Analytics (SecurityRecommendation, SecurityNestedRecommendation tables) and/or Event Hubs.
  • Security alerts → your SIEM.
  • Secure score and regulatory compliance snapshots → trend reporting.

Terraform:

resource "azurerm_security_center_automation" "export_findings" {
  name                = "export-findings-to-law"
  location            = azurerm_resource_group.security.location
  resource_group_name = azurerm_resource_group.security.name
  scopes              = ["/subscriptions/${var.subscription_id}"]

  action {
    type        = "LogAnalytics"
    resource_id = azurerm_log_analytics_workspace.security.id
  }

  source {
    event_source = "Assessments"
  }
  source {
    event_source = "SubAssessments"
  }
  source {
    event_source = "AssessmentsSnapshot"
  }
  source {
    event_source = "SubAssessmentsSnapshot"
  }
}

Snapshots give you a weekly full state; the non-snapshot sources stream changes as they happen. Deploy the export per subscription (or via the built-in Deploy export to Log Analytics workspace for Microsoft Defender for Cloud data policy at management-group scope).


Alerting on new findings

Column names in exported tables have shifted over time; check the schema in your workspace (SecurityNestedRecommendation | getschema) and adjust.

New critical/high server or container findings in the last hour:

SecurityNestedRecommendation
| where TimeGenerated > ago(1h)
| extend sev = tostring(coalesce(RecommendationSeverity, parse_json(AdditionalData).severity))
| where RecommendationState == "Unhealthy" and sev in ("High", "Critical")
| summarize firstSeen = min(TimeGenerated), resources = dcount(AssessedResourceId)
    by Finding = RecommendationName, sev
| order by sev asc, resources desc

KEV join — keep a small KEV_CL custom table refreshed daily (Logic App or Function pulling the CISA JSON), then:

SecurityNestedRecommendation
| where TimeGenerated > ago(1d) and RecommendationState == "Unhealthy"
| extend cve = extract(@"(CVE-\d{4}-\d{4,})", 1, strcat(RecommendationName, " ", tostring(AdditionalData)))
| where isnotempty(cve)
| join kind=inner (KEV_CL | project cve = cveID_s, dueDate = dueDate_s) on cve
| summarize resources = make_set(AssessedResourceId, 50) by cve, dueDate

Create these as Azure Monitor scheduled log search alerts with an action group that routes by owner (see Alert design). For "something changed" signals without polling, Defender for Cloud workflow automation can trigger a Logic App directly when a recommendation or alert fires — useful for creating tickets automatically.

You can also query Azure Resource Graph from a log search alert using arg("").securityresources, avoiding export latency for current-state checks.


Detecting regressions

A regression is a finding that was closed and came back. Common causes:

  • Autoscaling or redeploys from an old image or golden image.
  • Package downgraded or pinned by a config change.
  • A configuration baseline reverted by a vendor installer or manual change.
  • A new deployment of an old app version (rollback without re-fixing).

Detection approaches:

  1. Tracker state machine — your normalised finding store marks Closed – Remediated → Reopened when the same (asset, CVE, component) reappears. Alert on every reopen for P1/P2.
  2. Scanner-native — Tenable and Qualys both expose Reopened/Re-Opened states; include them in exports and alerts.
  3. Log Analytics comparison — compare the latest snapshot with the previous week's:
let thisWeek = SecurityNestedRecommendation
  | where TimeGenerated > ago(7d) and RecommendationState == "Unhealthy"
  | distinct AssessedResourceId, RecommendationName;
let lastWeekHealthy = SecurityNestedRecommendation
  | where TimeGenerated between (ago(14d) .. ago(7d)) and RecommendationState == "Healthy"
  | distinct AssessedResourceId, RecommendationName;
thisWeek
| join kind=inner lastWeekHealthy on AssessedResourceId, RecommendationName
| project AssessedResourceId, RecommendationName
  1. Image digest drift — alert when a running pod's image digest isn't one produced by the pipeline in the last N days (indicates an old image resurfacing).
  2. Golden image age — alert when the newest image version in the Compute Gallery is older than ~35 days.

Patch compliance and drift alerts

Azure Update Manager — log alert using Resource Graph:

arg("").patchassessmentresources
| where type endswith "/patchassessmentresults"
| extend critical = toint(properties.availablePatchCountByClassification.critical),
         security = toint(properties.availablePatchCountByClassification.security),
         lastAssessed = todatetime(properties.lastModifiedDateTime)
| where (critical + security) > 0 and lastAssessed > ago(2d)
| extend machine = tostring(split(id, "/patchAssessmentResults/")[0])
| project machine, critical, security

Suggested alerts:

Alert Condition Severity
Patch run failed Any patchinstallationresults with status Failed in last 24 h Sev 2
Stale assessment lastModifiedDateTime older than 3 days Sev 3
Reboot pending too long rebootPending == true for > 7 days Sev 3
Security updates outstanding past SLA Pending security updates on a machine for > 30 days (track first-seen in a custom table) Sev 2
Arc agent disconnected microsoft.hybridcompute/machines with status != Connected for > 24 h Sev 3

Configuration drift — Azure Policy compliance state changes are published to Event Grid (Microsoft.PolicyInsights.PolicyStateChanged). Subscribe a Function/Logic App filtered to your baseline assignments to alert when a machine goes non-compliant, rather than waiting for a report. For high-signal changes (admin group membership, security agent stopped), alert from Change Tracking / SIEM as described in Configuration Management.


Kubernetes in-cluster scanning with Prometheus

Outside Azure — or alongside Defender in a homelab or non-AKS cluster — Trivy Operator continuously scans workloads and exposes results as CRDs and Prometheus metrics.

helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm upgrade --install trivy-operator aqua/trivy-operator \
  --namespace trivy-system --create-namespace \
  --version <pinned-chart-version> \
  --set serviceMonitor.enabled=true \
  --set trivy.ignoreUnfixed=true

(Pin and verify chart and image versions — the same supply-chain lessons apply.)

kubectl get vulnerabilityreports -A -o wide
kubectl get configauditreports -A -o wide
kubectl get exposedsecretreports -A

Alerting rules:

groups:
  - name: vulnerability-management
    rules:
      - alert: CriticalVulnerabilitiesInRunningImage
        expr: sum by (namespace, image_repository, image_tag) (trivy_image_vulnerabilities{severity="Critical"}) > 0
        for: 1h
        labels: { severity: warning }
        annotations:
          summary: "Critical CVEs in {{ $labels.namespace }}/{{ $labels.image_repository }}:{{ $labels.image_tag }}"
          runbook_url: "https://docs.example.com/docs/vulnerability-management/container-and-application-remediation/"

      - alert: ExposedSecretInImage
        expr: sum by (namespace, image_repository) (trivy_image_exposedsecrets) > 0
        for: 15m
        labels: { severity: critical }
        annotations:
          summary: "Secret found in image {{ $labels.image_repository }} ({{ $labels.namespace }})"

For hosts, the node_exporter textfile collector scripts (apt_info, yum) expose pending updates and reboot-required metrics; see the Linux unattended upgrades guide for details.


Alert design

Alerts that go to a shared mailbox get ignored. Design for the owner.

Principle How
Route to owners Resolve the resource's owner tag (or CODEOWNERS for images) and route via action groups / Alertmanager receivers per team
Aggregate One alert per (finding, team) per day, listing affected resources — not one per resource
Severity follows priority P1/KEV → page; P2 → ticket + channel; P3/P4 → weekly digest
Every alert has a runbook link runbook_url annotation / alert description linking to the procedure
SLA-aware Alert at 75% of SLA elapsed, not just at creation
Self-resolving Alerts close when the condition clears (verification)
Measure noise Track alerts per team per week and the % acted on; prune or tune alerts nobody acts on

Dashboards

One dashboard per audience:

Engineering teams (daily) - My open findings by priority and due date - Findings breaching SLA in the next 7 days - My images with fixable HIGH/CRITICAL - Reopened findings

Platform / remediation owner (daily/weekly) - New findings (24 h, 7 d) by source - KEV exposure count and age - Patch compliance by ring and environment - Failed patch runs, stale assessments, disconnected agents - Pipeline gate failures and break-glass uses

Leadership (monthly) - SLA adherence trend by priority - MTTR trend - Open KEV count - Exceptions: count, age, expired - Coverage: % assets scanned

Azure Workbooks over Log Analytics + Resource Graph cover the Azure side; Grafana works well if you're also pulling Prometheus metrics and scanner exports into the same place.


References

  • Trivy documentation and GitHub advisory GHSA-69fq-xp46-6x23 (March 2026 compromise)
  • CVE-2026-33634 — NVD
  • GitHub Actions security hardening: pinning actions to a full-length commit SHA — docs.github.com
  • Defender for Cloud continuous export — Microsoft Learn
  • Defender for Cloud DevOps security — Microsoft Learn
  • Azure Monitor log search alerts with Azure Resource Graph — Microsoft Learn
  • Trivy Operator — aquasecurity.github.io/trivy-operator
  • Notary Project / Notation and AKS image integrity — Microsoft Learn