Andrew Mercer
on this page

The shape of a Terraform pipeline

flowchart LR
    subgraph MR["Merge request"]
      V[validate<br/>fmt · validate · tflint · scan] --> P[plan per env<br/>save plan · summary · policy]
    end
    subgraph Main["main branch"]
      AD[apply dev<br/>auto] --> AT[apply test<br/>manual] --> AP[apply prod<br/>manual + approval]
    end
    P -->|review + merge| AD

Non-negotiables:

  1. Plan on every MR, for every environment the change touches, and show the result on the MR.
  2. Apply only from the protected branch, through the pipeline.
  3. Apply what was reviewed: the saved plan, or a re-plan that's checked against it.
  4. One run per state at a time.
  5. Per-environment identity, prod credentials only in prod jobs.

A complete GitLab CI pipeline

For a directory-per-environment repo (live/<env>/<region>/<stack>). Terragrunt users: see the pipeline in the Terragrunt guide.

# .gitlab-ci.yml
default:
  image:
    name: hashicorp/terraform:1.16       # pin; or an internal image with tflint/trivy/conftest baked in
    entrypoint: [""]
  cache:
    key: tf-plugins-$CI_COMMIT_REF_SLUG
    paths: [.terraform.d/plugin-cache]

variables:
  TF_IN_AUTOMATION: "true"
  TF_INPUT: "false"
  TF_PLUGIN_CACHE_DIR: ${CI_PROJECT_DIR}/.terraform.d/plugin-cache
  ARM_USE_OIDC: "true"
  ARM_TENANT_ID: $AZURE_TENANT_ID

stages: [validate, plan, apply]

# ─── Reusable templates ──────────────────────────────────────────────────────
.azure_oidc:
  id_tokens:
    ARM_OIDC_TOKEN:
      aud: api://AzureADTokenExchange
  before_script:
    - mkdir -p "$TF_PLUGIN_CACHE_DIR"
    - cd "$STACK_DIR"
    - terraform init -input=false

.plan:
  extends: .azure_oidc
  stage: plan
  script:
    - terraform plan -lock-timeout=5m -out=tfplan
    - terraform show -json tfplan > tfplan.json
    # Summary for GitLab's MR Terraform widget
    - >
      jq '{create: [.resource_changes[] | select(.change.actions | index("create"))] | length,
           update: [.resource_changes[] | select(.change.actions == ["update"])] | length,
           delete: [.resource_changes[] | select(.change.actions | index("delete"))] | length}'
      tfplan.json > plan-summary.json
    - terraform show -no-color tfplan > plan.txt
  artifacts:
    name: plan-$CI_JOB_NAME_SLUG
    expire_in: 7 days                 # plans contain state data: keep retention short
    access: developer                 # GitLab 16.11+: restrict artifact downloads
    paths:
      - $STACK_DIR/tfplan
      - $STACK_DIR/tfplan.json
      - $STACK_DIR/plan.txt
      - $STACK_DIR/.terraform.lock.hcl
    reports:
      terraform: $STACK_DIR/plan-summary.json

.apply:
  extends: .azure_oidc
  stage: apply
  script:
    - terraform apply -lock-timeout=5m tfplan   # the exact plan that was reviewed
  resource_group: $STACK_DIR                    # serialise applies per state

# ─── Validate ────────────────────────────────────────────────────────────────
validate:
  stage: validate
  script:
    - terraform fmt -check -recursive
    - |
      for d in $(find live -name '*.tf' -not -path '*/.terraform/*' -exec dirname {} \; | sort -u); do
        (cd "$d" && terraform init -backend=false -input=false >/dev/null && terraform validate) || exit 1
      done
    - tflint --init && tflint --recursive
    - trivy config --severity HIGH,CRITICAL --exit-code 1 live/

# ─── dev ─────────────────────────────────────────────────────────────────────
plan:dev:
  extends: .plan
  variables:
    STACK_DIR: live/dev/canadacentral/network
    ARM_CLIENT_ID: $DEV_PLAN_CLIENT_ID
    ARM_SUBSCRIPTION_ID: $DEV_SUBSCRIPTION_ID
  environment: { name: dev, action: prepare }
  rules:
    - changes: [live/dev/canadacentral/network/**/*, modules/**/*]

apply:dev:
  extends: .apply
  needs: ["plan:dev"]
  variables:
    STACK_DIR: live/dev/canadacentral/network
    ARM_CLIENT_ID: $DEV_APPLY_CLIENT_ID
    ARM_SUBSCRIPTION_ID: $DEV_SUBSCRIPTION_ID
  environment: { name: dev }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes: [live/dev/canadacentral/network/**/*, modules/**/*]

# ─── prod ────────────────────────────────────────────────────────────────────
plan:prod:
  extends: .plan
  variables:
    STACK_DIR: live/prod/canadacentral/network
    ARM_CLIENT_ID: $PROD_PLAN_CLIENT_ID
    ARM_SUBSCRIPTION_ID: $PROD_SUBSCRIPTION_ID
  environment: { name: prod, action: prepare }
  rules:
    - changes: [live/prod/canadacentral/network/**/*, modules/**/*]

apply:prod:
  extends: .apply
  needs: ["plan:prod", "apply:dev"]
  variables:
    STACK_DIR: live/prod/canadacentral/network
    ARM_CLIENT_ID: $PROD_APPLY_CLIENT_ID
    ARM_SUBSCRIPTION_ID: $PROD_SUBSCRIPTION_ID
  environment: { name: prod }      # protected environment with required approvals
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes: [live/prod/canadacentral/network/**/*, modules/**/*]
      when: manual

Scaling it:

  • With many stacks, generate the jobs: a small script (or your own CLI) emits a child pipeline YAML with one plan/apply pair per changed stack, triggered with trigger: include: artifact. Or use parallel: matrix: with STACK_DIR values.
  • Put the templates in a CI component or shared include so 50 repos share one definition. GitLab's own Terraform templates were deprecated in favour of the OpenTofu CI/CD component (gitlab.com/components/opentofu); if you're on Terraform, maintain your own component.
  • rules: changes: keeps pipelines fast, but changes to shared modules or root-level config should trigger every dependent stack.

Saved plans vs re-planning

Apply the saved plan Re-plan at apply time
Guarantee Exactly what was reviewed What's true now
Stale plan Fails with "Saved plan is stale" if state changed since. Re-run the plan job Silently includes changes made since review
Artifacts Plan file (contains secrets) must move between jobs None
Terragrunt run --all Awkward (plans live in cache dirs) Natural

Prefer saved plans. If you re-plan, compare its summary to the reviewed one and fail on differences.

Plan files are tied to the same Terraform version, provider versions (ship the lock file) and working directory layout.

Concurrency and locking

  • GitLab resource_group: jobs with the same resource group run one at a time, queued in order. Use the state's directory or key as the group so unrelated stacks still run in parallel.
  • -lock-timeout=5m so a job waits for a briefly held lock instead of failing.
  • Never run force-unlock from automation.
  • Stale saved plans after a concurrent apply are a feature: the pipeline refuses to apply a plan computed against old state.

Identity per job

OIDC, per environment, plan vs apply (security):

Job Identity Roles
plan:dev (any branch) dev-plan Reader + state blob read/write (plan takes a lock) + data-plane reads needed by data sources
apply:dev (main) dev-apply Contributor on dev resource groups
plan:prod (any branch) prod-plan Reader on prod + prod state access
apply:prod (main, protected env) prod-apply Scoped write on prod

Federated credentials restrict which branch/environment each identity can be used from. MR pipelines from forks get no protected variables.

Drift detection

A scheduled pipeline per environment:

drift:prod:
  extends: .azure_oidc
  stage: plan
  variables:
    STACK_DIR: live/prod/canadacentral/network
    ARM_CLIENT_ID: $PROD_PLAN_CLIENT_ID
    ARM_SUBSCRIPTION_ID: $PROD_SUBSCRIPTION_ID
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  script:
    - set +e
    - terraform plan -detailed-exitcode -lock=false -input=false -no-color > drift.txt
    - rc=$?
    - set -e
    - if [ "$rc" -eq 2 ]; then echo "DRIFT DETECTED"; cat drift.txt; exit 1; fi
    - exit $rc
  artifacts:
    when: always
    paths: [$STACK_DIR/drift.txt]
    expire_in: 7 days

-lock=false keeps drift checks from blocking real applies; the plan is read-only. The failing job triggers GitLab's pipeline-failure notifications. The same pattern works on TeamCity as a scheduled build.

Planning across many repositories

When dozens of repos each have their own root modules, "what would change if…" means cloning and planning each. Tiers of solution:

  1. A script with a manifest: a list of repo/dir/env entries, git clone --depth 1, terraform plan -detailed-exitcode, run in parallel with xargs -P or GNU parallel, summarise exit codes. Works today, no infrastructure.
  2. CI fan-out: a "plan everything" pipeline in a central repo that triggers each repo's plan job via the API or multi-project pipelines, collecting results.
  3. A purpose-built tool: Atlantis, Terrateam, Digger, or a TACOS platform, which plan every affected root on every MR across repos.

Atlantis

Atlantis is a self-hosted server that runs Terraform in response to merge request webhooks and comments:

MR opened ─► Atlantis plans affected projects ─► comments plan on the MR
reviewer approves ─► someone comments "atlantis apply" ─► Atlantis applies, comments result
# atlantis.yaml in the repo
version: 3
automerge: false
parallel_plan: true
projects:
  - name: dev-network
    dir: live/dev/canadacentral/network
    workflow: azure
    autoplan:
      when_modified: ["*.tf", "../../../../modules/**/*.tf"]
  - name: prod-network
    dir: live/prod/canadacentral/network
    workflow: azure
    apply_requirements: [approved, mergeable]

workflows:
  azure:
    plan:
      steps:
        - init
        - plan
        - run: terraform show -json $PLANFILE > $SHOWFILE
    apply:
      steps:
        - apply

Strengths: locks per project across MRs (so two MRs can't plan-then-apply the same stack), plan visible where review happens, Conftest policy checks built in. Costs: a server to run and secure (it holds powerful credentials), and an apply-before-merge workflow some teams find odd. Run it on AKS with workload identity per environment, behind your ingress, with webhook secrets.

TACOS: managed platforms

"Terraform Automation and Collaboration Software":

Platform Notes
HCP Terraform / Terraform Enterprise Workspaces, remote runs, Sentinel/OPA/tfpolicy, Stacks, private registry, drift and health assessments
Spacelift Multi-IaC (Terraform, OpenTofu, Terragrunt, Pulumi, Ansible), OPA policies, stack dependencies
env0 Cost estimation, TTL environments, approval policies
Scalr HCP-compatible API, OpenTofu-first beyond Terraform 1.5.7
Terrateam, Digger Run in your own CI (GitHub-centric), lighter weight
Terragrunt Scale / Pipelines Gruntwork's offering for Terragrunt estates

Buy one when the platform team's time maintaining pipeline glue costs more than the licence, or when you need features like policy sets and audit trails out of the box.

Other CI systems

  • GitHub Actions: hashicorp/setup-terraform or opentofu/setup-opentofu, environments with required reviewers, OIDC via azure/login.
  • TeamCity: same stages as build configurations; snapshot dependencies for ordering; a shared build template with the init/plan/apply steps; personal builds to plan a branch without pushing.
  • Azure DevOps: environments with approvals and checks, workload identity federation service connections.

The principles don't change: plan on review, apply what was reviewed, serialise per state, scope identities per environment.