Terraform architectures

Key points:

• A state file is a blast radius - split state by environment, layer and ownership.
• Use a folder per environment for environments that differ; workspaces only for many identical copies.
• Share data between stacks with data sources or a parameter store, not terraform_remote_state.
• In CI, apply the reviewed plan file, authenticate with OIDC, and alert on drift.
• Use for_each over count, moved blocks for renames, and prevent_destroy on what can’t be rebuilt.

State is the architecture

A state file is a blast radius: everything in it is planned, locked and applied together. One bad apply touches anything in it; one lock blocks everyone.

Split state along three lines:

1. Environment - prod never shares state with dev.
2. Layer - by change speed: network yearly, cluster monthly, app infrastructure weekly.
3. Ownership - a team can apply its stack without locking anyone else’s.

flowchart TB
    NET["network<br/>VPC, subnets, NAT"] --> CLUSTER["cluster<br/>EKS, node groups, IAM"]
    CLUSTER --> PLATFORM["platform<br/>DNS, certificates, secrets, add-ons"]
    PLATFORM --> APPS["app infrastructure<br/>databases, buckets, queues"]

Each box is its own state; an app team’s apply can’t touch the VPC.

Directory layout

modules/                    # reusable, versioned building blocks
├── vpc/
├── eks-cluster/
└── rds-postgres/
live/                       # one folder = one state
├── prod/
│   ├── network/
│   ├── cluster/
│   └── payments-db/
└── staging/
    ├── network/
    ├── cluster/
    └── payments-db/

Give each live/ folder its own backend key, calling modules with environment-specific inputs.

Directories vs workspaces

	Folder per environment	Workspaces
Code	Can differ between environments	Identical for every environment
State	Separate backend key per folder	One backend, one state per workspace
Credentials	Easy to use a different account per environment	Same backend config for all
Risk	Visible in the path you’re in	Easy to apply to the wrong workspace
Best for	Environments in separate accounts that genuinely differ	Many identical copies, e.g. per-customer stacks

Sharing data between stacks

Prefer data sources or a parameter store - they let you refactor one stack without breaking others.

• Data sources - look up by tag or name (data "aws_vpc" with a tag filter). No coupling to the other stack’s code.
• A parameter store - the producer writes IDs to SSM Parameter Store, consumers read them. A clear, minimal contract.
• Avoid terraform_remote_state. Quick, but it needs read access to the other whole state file, secrets included, and couples stacks tightly.

Modules

• Small and focused. A “whole environment” module can’t be reused or reviewed.
• No provider blocks inside modules. Pass providers from the root so the caller controls region, account and credentials.
• Version them. Pin to a tag (?ref=v1.4.0) or registry version so a module change doesn’t silently hit every environment.
• Pin provider versions with ~> constraints, and commit .terraform.lock.hcl.

Running it from CI

1. On pull request: terraform plan -out=tfplan, posted as a comment for review.
2. On merge: terraform apply tfplan - apply the plan that was reviewed, not a fresh one.
3. Credentials: OIDC from CI to the cloud, no stored keys. See the Terraform snippets.
4. Drift detection: scheduled terraform plan -detailed-exitcode. Exit code 2 means reality no longer matches the code - alert on it.

Gotchas

• count shifts indexes. Removing the first count item destroys and recreates everything after it. Use for_each with stable keys.
• Renames destroy. Renaming or moving a resource into a module looks like delete-and-create. Add a moved block; confirm the plan shows a move.
• State contains secrets. Passwords and private keys land in state in plain text. Encrypt the backend, restrict who can read it, and treat state access like production access.
• Cluster and contents in one apply fails. The Kubernetes and Helm providers need the cluster to exist before they can be configured. Put the cluster and its contents in separate states.
• Externally changed fields diff forever. An autoscaler sets a node group’s desired size; Terraform reverts it every apply. Use lifecycle { ignore_changes = [...] } for fields another system owns.
• -target is a smell. Needing it regularly means the state is too big or tangled. Split it.
• depends_on on a module defers its data sources. They resolve at apply, turning a clean plan into a wall of “known after apply”. Depend on specific resources or outputs instead.
• Import, don’t recreate. Resources that already exist aren’t in state yet. Use an import block (Terraform 1.5+) and plan until the diff is empty.
• Unprotected irreplaceable resources. Databases and state buckets can’t be rebuilt after an accidental destroy. Set prevent_destroy plus deletion protection on the resource itself.