← Lessons/State — Break It & Recover

State — Break It & Recover

This is the lesson that separates people who use Terraform from people who understand it. You're going to deliberately break Terraform's state in the three ways it breaks in real life — drift, a deleted state file, and a concurrent write — and recover from each. Safely, because this is a simulator.

Each mini lesson builds on the state the one before it left, so they unlock in order. The buttons beside the State Inspector (edit in the console, delete the state file, import, two applies at once) appear on the steps that use them.

Golden rule: never hand-edit or delete state, and never let two applies share one local state file. The buttons let you do exactly that — so you feel why the rule exists.

statedriftimportstate-recoverytroubleshooting

Interview prep

State is the #1 source of Terraform interview questions and real-world incidents.

What is configuration drift and how do you detect it?

Drift is when real infrastructure changes outside Terraform (a console edit, a hotfix, another tool), so reality no longer matches state. You detect it by running terraform plan (or plan -refresh-only): the refresh step reads the real world and the plan surfaces the difference. You resolve it by either applying to bring reality back to your code, or updating your code to match.

Why should you never edit or delete the state file by hand?

State is the only mapping between your config and real resource IDs. Delete it and Terraform forgets what it owns — the next apply tries to recreate resources that already exist (duplicates/conflicts), and the originals become orphaned. Hand-editing risks the same corruption. To re-establish the mapping you use terraform import, not a text editor.

Two engineers run terraform apply at the same time against the same state. What happens, and how do you prevent it?

Without locking, concurrent writes interleave and corrupt the state file (serial mismatch / lost updates). You prevent it with a remote backend that supports state locking (e.g. S3 + DynamoDB, Terraform Cloud, GCS), so the second apply waits for the lock instead of racing.