I moved the state file and ended up with two identical servers

Goal

Open up the structure of a state file, and cause and confirm one by one, with OpenTofu, duplicate creation when state is lost, the lineage safeguard, import recovery, the concurrent-apply lock, state splitting, and commit rules.

Why it matters

State is the only record that connects the code to the actual infrastructure, so accidents usually come from the state, not the code. If the state disappears, the tool does not know what already exists and creates it again, and if you overwrite with the wrong state, you erase someone else's record. The only recovery is to bring things back one by one with import, so a versioned backend is the baseline. A lock is what keeps two applies from overlapping, and you must not keep layers with different lifetimes in one state, so that what changes daily does not put at risk what almost never changes.

Steps

In /root/iac-state/main.tf, put random_id.server (byte_length 4) and one more resource, which writes to out/server-<그 hex>.txt the line server <hex> (the placeholders stand for that hex value), namely local_file.inventory, and run init and apply. Then copy the state file to /root/iac-state/backup/terraform.tfstate, and using the values read from that copy, write three lines to /root/iac-state/anatomy.txt: lineage=<값>, serial=<값>, and resources=<리소스 블록 수> (the placeholders are the value and the number of resource blocks).
Create /root/iac-state/lost/ and move terraform.tfstate and terraform.tfstate.backup (if present) there. Save the output of tofu plan to /root/iac-state/lost-plan.txt and then apply. The grader checks that the moved state and the new state have different lineages, and that two server files have appeared in out/.
Try to put the old state back with tofu state push lost/terraform.tfstate. Save the output (including the error) to /root/iac-state/push.txt. Do not use -force. The grader also checks whether the current state's lineage is still different from the old state's.
Copy into /root/iac-state/rebuild/ the same main.tf, and use an import block to bring in, from the old state (lost/terraform.tfstate), the random_id.server. Find the import identifier in that resource's attributes in the old state. After init and apply in rebuild, the plan must be clean. The grader checks whether the hex of the rebuild state is the same as the hex of the old server.
In /root/iac-state/lockdemo/main.tf, put terraform_data.slow, which sleeps for 15 seconds with local-exec, and run init. Start apply in the background, run tofu plan in the same directory before it finishes, and save the output to /root/iac-state/lock.txt. Then run tofu plan -lock-timeout=60s, wait until the lock is released, and save the output afterward to /root/iac-state/lockwait.txt. The grader also checks by creating the same situation itself in a temporary directory.
In /root/iac-state/network/, put random_id.vpc (byte_length 3) and the output vpc_id = "vpc-<hex>"; in /root/iac-state/app/, put a terraform_remote_state data source (local backend, ../network/terraform.tfstate) that reads that output, and a resource that writes to out/app.conf the line vpc=<vpc_id>, namely local_file.app. Run init and apply network first and app afterward. The plan must be clean in both directories.
Create /root/iac-state/.gitignore. By the git check-ignore criterion, terraform.tfstate, terraform.tfstate.backup, lost/terraform.tfstate, .terraform/ (the files inside it), and plan files such as change.tfplan must be ignored, and .terraform.lock.hcl, main.tf, and app/main.tf must not be ignored. The grader checks by placing this .gitignore in a temporary git repository.

Notes

The Pod has OpenTofu 1.9.0 and local and random provider mirrors, so it runs without internet. All the backends in this lab are local.
The local backend's lock is an operating-system file lock, so if the process that is applying dies, the lock is released too. So the "leftover lock" that needs a force-unlock arises only with remote backends and is not reproduced in this Pod.
Common mistake: leaving terraform.tfstate.backup in place in step 2 — the tool does not automatically restore just because a backup exists, but it makes it confusing later which one is the real one.
Common mistake: putting the hex in the identifier when recovering with import. The identifier differs for each resource, so check the provider documentation.
State · State Locking · tofu state push · Import · random_id (Import) · terraform_remote_state · Backend: local

Open up the state file

In /root/iac-state/main.tf, put random_id.server (byte_length 4) and one more resource, which writes to out/server-<그 hex>.txt the line server <hex> (the placeholders stand for that hex value), namely local_file.inventory, and run init and apply. Then copy the state file to /root/iac-state/backup/terraform.tfstate, and using the values read from that copy, write three lines to /root/iac-state/anatomy.txt: lineage=<값>, serial=<값>, and resources=<리소스 블록 수> (the placeholders are the value and the number of resource blocks).

The state file is JSON. The lineage is a unique number attached when this state was first created, and the serial goes up every time it is written. With jq, look at .lineage, .serial, and .resources | length.

We lost the state and got one more of the same server

Create /root/iac-state/lost/ and move terraform.tfstate and terraform.tfstate.backup (if present) there. Save the output of tofu plan to /root/iac-state/lost-plan.txt and then apply. The grader checks that the moved state and the new state have different lineages, and that two server files have appeared in out/.

The tool sees a resource not in the state as "not yet created." Because it is a resource whose name is chosen at random, one more quietly appears without a collision. Read the Plan line of the plan output.

We tried to overwrite with the old state and it was rejected

Try to put the old state back with tofu state push lost/terraform.tfstate. Save the output (including the error) to /root/iac-state/push.txt. Do not use -force. The grader also checks whether the current state's lineage is still different from the old state's.

Two states with different lineages are not "different points in time of the same infrastructure" but "two separate records." Think about why the tool prevents this. -force is the option that turns off this safeguard.

Get the lost server back with import

Copy into /root/iac-state/rebuild/ the same main.tf, and use an import block to bring in, from the old state (lost/terraform.tfstate), the random_id.server. Find the import identifier in that resource's attributes in the old state. After init and apply in rebuild, the plan must be clean. The grader checks whether the hex of the rebuild state is the same as the hex of the old server.

random_id supports import and takes the b64_url value as the identifier (see the provider documentation). You need to use only the two items to and id in the import block, and the plan must show "will be imported." If a new random_id is being created, it is wrong.

While an apply runs, other plans are blocked by the lock

In /root/iac-state/lockdemo/main.tf, put terraform_data.slow, which sleeps for 15 seconds with local-exec, and run init. Start apply in the background, run tofu plan in the same directory before it finishes, and save the output to /root/iac-state/lock.txt. Then run tofu plan -lock-timeout=60s, wait until the lock is released, and save the output afterward to /root/iac-state/lockwait.txt. The grader also checks by creating the same situation itself in a temporary directory.

Every command that can write the state takes the lock automatically. By default it does not wait and fails at once, and if you give -lock-timeout, it waits that long. Start a background job with & and wait for its end with wait.

Split the state for layers with different lifetimes and connect them only through outputs

In /root/iac-state/network/, put random_id.vpc (byte_length 3) and the output vpc_id = "vpc-<hex>"; in /root/iac-state/app/, put a terraform_remote_state data source (local backend, ../network/terraform.tfstate) that reads that output, and a resource that writes to out/app.conf the line vpc=<vpc_id>, namely local_file.app. Run init and apply network first and app afterward. The plan must be clean in both directories.

The app state holds no network resources, only the data source. So even if you change app every day, the network is not locked and does not get into the plan. Instead, app comes to depend on the output name of network.

Do not commit state and plans, but do commit the lock file

Create /root/iac-state/.gitignore. By the git check-ignore criterion, terraform.tfstate, terraform.tfstate.backup, lost/terraform.tfstate, .terraform/ (the files inside it), and plan files such as change.tfplan must be ignored, and .terraform.lock.hcl, main.tf, and app/main.tf must not be ignored. The grader checks by placing this .gitignore in a temporary git repository.

Values go into the state and saved plans in plain text (you will see it directly in the next lab). The lock file pins provider versions and hashes, so it must be reviewed together with the code. Check how far a pattern like *.tfstate.* reaches.