The shape of a Terraform pipeline¶
flowchart LR
subgraph MR["Merge request"]
V[validate<br/>fmt · validate · tflint · scan] --> P[plan per env<br/>save plan · summary · policy]
end
subgraph Main["main branch"]
AD[apply dev<br/>auto] --> AT[apply test<br/>manual] --> AP[apply prod<br/>manual + approval]
end
P -->|review + merge| AD
Non-negotiables:
- Plan on every MR, for every environment the change touches, and show the result on the MR.
- Apply only from the protected branch, through the pipeline.
- Apply what was reviewed: the saved plan, or a re-plan that's checked against it.
- One run per state at a time.
- Per-environment identity, prod credentials only in prod jobs.
A complete GitLab CI pipeline¶
For a directory-per-environment repo (live/<env>/<region>/<stack>). Terragrunt users: see the pipeline in the Terragrunt guide.
# .gitlab-ci.yml
default:
image:
name: hashicorp/terraform:1.16 # pin; or an internal image with tflint/trivy/conftest baked in
entrypoint: [""]
cache:
key: tf-plugins-$CI_COMMIT_REF_SLUG
paths: [.terraform.d/plugin-cache]
variables:
TF_IN_AUTOMATION: "true"
TF_INPUT: "false"
TF_PLUGIN_CACHE_DIR: ${CI_PROJECT_DIR}/.terraform.d/plugin-cache
ARM_USE_OIDC: "true"
ARM_TENANT_ID: $AZURE_TENANT_ID
stages: [validate, plan, apply]
# ─── Reusable templates ──────────────────────────────────────────────────────
.azure_oidc:
id_tokens:
ARM_OIDC_TOKEN:
aud: api://AzureADTokenExchange
before_script:
- mkdir -p "$TF_PLUGIN_CACHE_DIR"
- cd "$STACK_DIR"
- terraform init -input=false
.plan:
extends: .azure_oidc
stage: plan
script:
- terraform plan -lock-timeout=5m -out=tfplan
- terraform show -json tfplan > tfplan.json
# Summary for GitLab's MR Terraform widget
- >
jq '{create: [.resource_changes[] | select(.change.actions | index("create"))] | length,
update: [.resource_changes[] | select(.change.actions == ["update"])] | length,
delete: [.resource_changes[] | select(.change.actions | index("delete"))] | length}'
tfplan.json > plan-summary.json
- terraform show -no-color tfplan > plan.txt
artifacts:
name: plan-$CI_JOB_NAME_SLUG
expire_in: 7 days # plans contain state data: keep retention short
access: developer # GitLab 16.11+: restrict artifact downloads
paths:
- $STACK_DIR/tfplan
- $STACK_DIR/tfplan.json
- $STACK_DIR/plan.txt
- $STACK_DIR/.terraform.lock.hcl
reports:
terraform: $STACK_DIR/plan-summary.json
.apply:
extends: .azure_oidc
stage: apply
script:
- terraform apply -lock-timeout=5m tfplan # the exact plan that was reviewed
resource_group: $STACK_DIR # serialise applies per state
# ─── Validate ────────────────────────────────────────────────────────────────
validate:
stage: validate
script:
- terraform fmt -check -recursive
- |
for d in $(find live -name '*.tf' -not -path '*/.terraform/*' -exec dirname {} \; | sort -u); do
(cd "$d" && terraform init -backend=false -input=false >/dev/null && terraform validate) || exit 1
done
- tflint --init && tflint --recursive
- trivy config --severity HIGH,CRITICAL --exit-code 1 live/
# ─── dev ─────────────────────────────────────────────────────────────────────
plan:dev:
extends: .plan
variables:
STACK_DIR: live/dev/canadacentral/network
ARM_CLIENT_ID: $DEV_PLAN_CLIENT_ID
ARM_SUBSCRIPTION_ID: $DEV_SUBSCRIPTION_ID
environment: { name: dev, action: prepare }
rules:
- changes: [live/dev/canadacentral/network/**/*, modules/**/*]
apply:dev:
extends: .apply
needs: ["plan:dev"]
variables:
STACK_DIR: live/dev/canadacentral/network
ARM_CLIENT_ID: $DEV_APPLY_CLIENT_ID
ARM_SUBSCRIPTION_ID: $DEV_SUBSCRIPTION_ID
environment: { name: dev }
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
changes: [live/dev/canadacentral/network/**/*, modules/**/*]
# ─── prod ────────────────────────────────────────────────────────────────────
plan:prod:
extends: .plan
variables:
STACK_DIR: live/prod/canadacentral/network
ARM_CLIENT_ID: $PROD_PLAN_CLIENT_ID
ARM_SUBSCRIPTION_ID: $PROD_SUBSCRIPTION_ID
environment: { name: prod, action: prepare }
rules:
- changes: [live/prod/canadacentral/network/**/*, modules/**/*]
apply:prod:
extends: .apply
needs: ["plan:prod", "apply:dev"]
variables:
STACK_DIR: live/prod/canadacentral/network
ARM_CLIENT_ID: $PROD_APPLY_CLIENT_ID
ARM_SUBSCRIPTION_ID: $PROD_SUBSCRIPTION_ID
environment: { name: prod } # protected environment with required approvals
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
changes: [live/prod/canadacentral/network/**/*, modules/**/*]
when: manual
Scaling it:
- With many stacks, generate the jobs: a small script (or your own CLI) emits a child pipeline YAML with one plan/apply pair per changed stack, triggered with
trigger: include: artifact. Or useparallel: matrix:withSTACK_DIRvalues. - Put the templates in a CI component or shared include so 50 repos share one definition. GitLab's own Terraform templates were deprecated in favour of the OpenTofu CI/CD component (
gitlab.com/components/opentofu); if you're on Terraform, maintain your own component. rules: changes:keeps pipelines fast, but changes to shared modules or root-level config should trigger every dependent stack.
Saved plans vs re-planning¶
| Apply the saved plan | Re-plan at apply time | |
|---|---|---|
| Guarantee | Exactly what was reviewed | What's true now |
| Stale plan | Fails with "Saved plan is stale" if state changed since. Re-run the plan job | Silently includes changes made since review |
| Artifacts | Plan file (contains secrets) must move between jobs | None |
Terragrunt run --all |
Awkward (plans live in cache dirs) | Natural |
Prefer saved plans. If you re-plan, compare its summary to the reviewed one and fail on differences.
Plan files are tied to the same Terraform version, provider versions (ship the lock file) and working directory layout.
Concurrency and locking¶
- GitLab
resource_group: jobs with the same resource group run one at a time, queued in order. Use the state's directory or key as the group so unrelated stacks still run in parallel. -lock-timeout=5mso a job waits for a briefly held lock instead of failing.- Never run
force-unlockfrom automation. - Stale saved plans after a concurrent apply are a feature: the pipeline refuses to apply a plan computed against old state.
Identity per job¶
OIDC, per environment, plan vs apply (security):
| Job | Identity | Roles |
|---|---|---|
plan:dev (any branch) |
dev-plan | Reader + state blob read/write (plan takes a lock) + data-plane reads needed by data sources |
apply:dev (main) |
dev-apply | Contributor on dev resource groups |
plan:prod (any branch) |
prod-plan | Reader on prod + prod state access |
apply:prod (main, protected env) |
prod-apply | Scoped write on prod |
Federated credentials restrict which branch/environment each identity can be used from. MR pipelines from forks get no protected variables.
Drift detection¶
A scheduled pipeline per environment:
drift:prod:
extends: .azure_oidc
stage: plan
variables:
STACK_DIR: live/prod/canadacentral/network
ARM_CLIENT_ID: $PROD_PLAN_CLIENT_ID
ARM_SUBSCRIPTION_ID: $PROD_SUBSCRIPTION_ID
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
script:
- set +e
- terraform plan -detailed-exitcode -lock=false -input=false -no-color > drift.txt
- rc=$?
- set -e
- if [ "$rc" -eq 2 ]; then echo "DRIFT DETECTED"; cat drift.txt; exit 1; fi
- exit $rc
artifacts:
when: always
paths: [$STACK_DIR/drift.txt]
expire_in: 7 days
-lock=false keeps drift checks from blocking real applies; the plan is read-only. The failing job triggers GitLab's pipeline-failure notifications. The same pattern works on TeamCity as a scheduled build.
Planning across many repositories¶
When dozens of repos each have their own root modules, "what would change if…" means cloning and planning each. Tiers of solution:
- A script with a manifest: a list of repo/dir/env entries,
git clone --depth 1,terraform plan -detailed-exitcode, run in parallel withxargs -Por GNU parallel, summarise exit codes. Works today, no infrastructure. - CI fan-out: a "plan everything" pipeline in a central repo that triggers each repo's plan job via the API or multi-project pipelines, collecting results.
- A purpose-built tool: Atlantis, Terrateam, Digger, or a TACOS platform, which plan every affected root on every MR across repos.
Atlantis¶
Atlantis is a self-hosted server that runs Terraform in response to merge request webhooks and comments:
MR opened ─► Atlantis plans affected projects ─► comments plan on the MR
reviewer approves ─► someone comments "atlantis apply" ─► Atlantis applies, comments result
# atlantis.yaml in the repo
version: 3
automerge: false
parallel_plan: true
projects:
- name: dev-network
dir: live/dev/canadacentral/network
workflow: azure
autoplan:
when_modified: ["*.tf", "../../../../modules/**/*.tf"]
- name: prod-network
dir: live/prod/canadacentral/network
workflow: azure
apply_requirements: [approved, mergeable]
workflows:
azure:
plan:
steps:
- init
- plan
- run: terraform show -json $PLANFILE > $SHOWFILE
apply:
steps:
- apply
Strengths: locks per project across MRs (so two MRs can't plan-then-apply the same stack), plan visible where review happens, Conftest policy checks built in. Costs: a server to run and secure (it holds powerful credentials), and an apply-before-merge workflow some teams find odd. Run it on AKS with workload identity per environment, behind your ingress, with webhook secrets.
TACOS: managed platforms¶
"Terraform Automation and Collaboration Software":
| Platform | Notes |
|---|---|
| HCP Terraform / Terraform Enterprise | Workspaces, remote runs, Sentinel/OPA/tfpolicy, Stacks, private registry, drift and health assessments |
| Spacelift | Multi-IaC (Terraform, OpenTofu, Terragrunt, Pulumi, Ansible), OPA policies, stack dependencies |
| env0 | Cost estimation, TTL environments, approval policies |
| Scalr | HCP-compatible API, OpenTofu-first beyond Terraform 1.5.7 |
| Terrateam, Digger | Run in your own CI (GitHub-centric), lighter weight |
| Terragrunt Scale / Pipelines | Gruntwork's offering for Terragrunt estates |
Buy one when the platform team's time maintaining pipeline glue costs more than the licence, or when you need features like policy sets and audit trails out of the box.
Other CI systems¶
- GitHub Actions:
hashicorp/setup-terraformoropentofu/setup-opentofu, environments with required reviewers, OIDC viaazure/login. - TeamCity: same stages as build configurations; snapshot dependencies for ordering; a shared build template with the init/plan/apply steps; personal builds to plan a branch without pushing.
- Azure DevOps: environments with approvals and checks, workload identity federation service connections.
The principles don't change: plan on review, apply what was reviewed, serialise per state, scope identities per environment.