Andrew Mercer
on this page
Page Covers
This page What state is, locking, the state CLI, drift, workspaces, security, recovery
Remote state and backends Backend configuration (azurerm, s3, gcs, GitLab HTTP), bootstrapping, partial config, reading other stacks
Refactoring moved, import, removed, terraform query, and the state commands for when blocks aren't enough

What state is

Terraform state is a JSON document that maps addresses in your configuration (azurerm_subnet.this["aks"]) to real objects (/subscriptions/.../subnets/aks), along with every attribute Terraform last saw for each object.

Terraform needs it because:

  1. Mapping. Cloud APIs don't know which Terraform resource created an object. State is the only link.
  2. Diffing. A plan compares configuration vs state (refreshed against reality) to decide what to change.
  3. Dependencies for destroy. When you delete a resource block, the configuration no longer says what it depended on. State remembers.
  4. Performance. Large estates can skip refresh (-refresh=false) and plan from state alone.

A peek inside:

{
  "version": 4,
  "terraform_version": "1.16.6",
  "serial": 42,
  "lineage": "8c1f0a6e-...",
  "outputs": { "vnet_id": { "value": "/subscriptions/.../vnet-app-dev-cc", "type": "string" } },
  "resources": [
    {
      "mode": "managed",
      "type": "azurerm_subnet",
      "name": "this",
      "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]",
      "instances": [
        { "index_key": "aks", "attributes": { "id": "...", "address_prefixes": ["10.10.0.0/22"] } }
      ]
    }
  ]
}
  • serial increments on every write. Backends reject writing an older serial.
  • lineage is a UUID fixed when the state is created. Terraform refuses to overwrite a state with a different lineage, which protects against pushing the wrong file.

Never edit state by hand. Use the CLI or, better, configuration blocks (refactoring).

State contains secrets

Every attribute of every resource is in state in plaintext, including:

  • Database admin passwords, storage account keys, connection strings
  • tls_private_key private keys, random_password results
  • Anything marked sensitive (sensitive only hides it in CLI output)

So:

  • Store state remotely, encrypted at rest, with tight access control. Read access to state is read access to every secret in it.
  • Keep secrets out of state where possible: ephemeral resources and write-only arguments (Terraform 1.11+, OpenTofu 1.11+), managed identities instead of keys, Key Vault references instead of values. See security.
  • OpenTofu can encrypt state client-side (1.7+) with keys from a KMS, so the backend only ever sees ciphertext. Terraform relies on backend-side encryption. See OpenTofu.
  • Plan files contain state too. Treat tfplan artifacts in CI exactly like state: short retention, restricted access.

Locking

When two runs write the same state at once, you get corruption or lost updates. Backends that support locking take a lock for any operation that could write:

Backend Locking mechanism
azurerm Blob lease (built in)
s3 use_lockfile = true (S3 conditional writes, GA in Terraform 1.11; OpenTofu 1.10+). DynamoDB is deprecated
gcs Built in
http (GitLab) Built in via lock/unlock endpoints
HCP Terraform Built in, per workspace
local OS file lock on the local machine only
terraform plan -lock-timeout=10m      # wait for a lock instead of failing immediately
terraform force-unlock <LOCK_ID>      # only when you're certain the holder is dead

force-unlock on a lock held by a running apply is how states get corrupted. Check the lock's Who and Created fields, check CI for a running job, then unlock. In CI, prevent concurrent runs per state with GitLab resource_group or equivalent (CI/CD).

The state CLI

Command Use
terraform state list List addresses (state list 'module.net.azurerm_subnet.this' filters)
terraform state show <addr> Show one resource's attributes (-json in 1.16+)
terraform state mv <from> <to> Rename an address. Prefer moved blocks
terraform state rm <addr> Forget a resource without destroying it. Prefer removed blocks
terraform import <addr> <id> Adopt an existing object. Prefer import blocks
terraform state pull > backup.tfstate Download raw state
terraform state push backup.tfstate Upload raw state (checks lineage/serial; -force skips, dangerously)
terraform state replace-provider <from> <to> Change provider source addresses (registry moves, forks)
terraform show / show -json Human / machine-readable state or plan
terraform output [-json] [name] Root module outputs

Why the block forms are preferred: moved, import and removed blocks go through code review and plan, and every environment gets the same change when it applies. CLI state surgery happens once, on one state, from one laptop, with no record. Details in refactoring.

Before any CLI state surgery: terraform state pull > pre-surgery-$(date +%F).tfstate.

Drift

Drift is any difference between real infrastructure and what state/configuration expects: someone clicked in the portal, Azure Policy added a tag, an autoscaler changed a node count.

# Show drift without proposing to "fix" it
terraform plan -refresh-only

# Accept the drift into state (configuration unchanged; next plan will try to revert
# unless you also update configuration)
terraform apply -refresh-only

# CI-friendly drift detection: exit 0 = no changes, 1 = error, 2 = changes
terraform plan -detailed-exitcode -lock=false -input=false

A drift-detection job is cheap and catches problems before the next real change surfaces them:

  • Schedule plan -detailed-exitcode nightly per stack (TeamCity, GitLab scheduled pipelines). Exit code 2 opens an issue or posts to chat.
  • Triage: revert (apply the configuration), adopt (change configuration to match), or ignore (ignore_changes with a comment, for attributes another system owns).
  • Recurring drift on the same attribute is a process problem, not a Terraform problem. Lock the portal down with RBAC or Azure Policy.

Workspaces

CLI workspaces give one configuration several independent states in the same backend:

terraform workspace new feature-x
terraform workspace select feature-x
terraform workspace list            # -json in 1.16+
locals {
  name_suffix = terraform.workspace == "default" ? "" : "-${terraform.workspace}"
}

Good for: short-lived copies of one environment (per-branch preview stacks, load-test copies).

Bad for dev/test/prod separation:

  • Same backend, so usually the same credentials for every environment
  • Nothing in the directory or prompt shows which workspace you're in
  • Code ends up full of terraform.workspace == "prod" ? ... : ...

Use separate directories, backends and credentials per environment instead. See multi-environment configuration. HCP Terraform "workspaces" are a different, much heavier concept: one workspace = one state + variables + permissions + run history.

Recovering from state problems

Problem Recovery
State file deleted or corrupted Restore a previous blob/object version. This is why backend versioning is non-negotiable.
Apply crashed mid-way Re-run plan. Terraform writes state as it goes (errored.tfstate locally if it couldn't write remotely: terraform state push errored.tfstate)
Stuck lock Confirm the holder is dead, then force-unlock
Resource exists in cloud but not state import block
Resource in state but deleted in cloud plan shows it will be created again; or removed block if you don't want it back
Wrong state pushed over the right one Restore previous version from backend versioning
Two configurations managing one resource Pick an owner; removed { destroy = false } in the other

With azurerm, turn on blob versioning and soft delete on the state storage account. With S3, bucket versioning. With GCS, object versioning. Test a restore once, before you need it.

How big should a state be?

  • Plan time scales with resource count because of refresh. Hundreds of resources is fine; a couple of thousand means multi-minute plans.
  • Blast radius scales with resource count. A typo in a tag can produce a plan that touches networking if networking shares the state.
  • Lock contention scales with the number of people changing it.

A useful target is a few dozen to a couple of hundred resources per state, split by layer and lifecycle (network, data, compute, apps). The architecture guide covers layering in detail.

Short-term speedups for an oversized state:

terraform plan -refresh=false        # trust state, skip API reads (miss drift)
terraform plan -parallelism=30       # more concurrent API calls (watch throttling)
terraform plan -target=module.app    # last resort; see the warning below

-target produces a partial plan that ignores dependencies outside the target. Use it for emergencies and recovery, never as a routine workflow. Terraform prints a warning for a reason.

in this section
* remote state and backends
* refactoring state: moved, import, removed and terraform query