Writing a Playbook That Stays Quiet on the Second Run

Goal

You create a state in which, no matter how many times the same playbook runs, nothing changes from the second run on, and you prove it with logs and a report.

Why it matters

Idempotency is not a matter of "pretty code" but the condition that makes automation actually usable. Only with confidence that the second run is safe can you schedule periodic runs from cron to catch drift and rerun fearlessly after a failure in the middle of a pipeline. By contrast, people avoid a playbook that restarts the service every time, and automation that is avoided is as good as no automation. The key mechanisms are modules that know the state, guards on shell commands (creates/changed_when), and handlers that run only when there is a change. In particular, remember that handlers run bunched together at the end of the play — if the play fails midway, the handlers vanish entirely, leaving "the configuration has changed but the service is looking at the old configuration".

Steps

Create /root/ans/idem/site.yml, run it, and save the output to /root/ans/idem/out/run1.txt. The changed of the first run must be 2 or more.
Run the same playbook again and save it to /root/ans/idem/out/run2.txt. This time it must be changed=0.
Define a handler called reload app and call it with notify from the task that handles the configuration. When the handler runs, it must create /root/ans/idem/artifacts/reload.marker.
The second run's log (run2.txt) must not contain RUNNING HANDLER. The first run's log must contain it.
Add one command/shell task and put a creates guard on it. That task writes one line to /root/ans/idem/artifacts/stamp.txt, and even if it runs twice, the file must contain only one line.
Use changed_when at least once on a task that only queries, and failed_when at least once on a task that needs its own failure decision.
Run the playbook with --check and save the output to /root/ans/idem/out/check.txt. It must be changed=0 and failed=0.
Create /root/ans/idem/out/idempotency.json. It holds four keys: run1_changed (2 or more), run2_changed (0), handler_fired (1 or true), and verdict ("idempotent").

Notes

A lab Pod is created fresh for each lab. If /root/ans/inventory/hosts.ini does not exist, first recreate the same inventory you made in the first lab (web1, web2, db1, ansible_host=127.0.0.1, ansible_port=2222, ansible_user=root, web and db under [prod:children]). For the structure, refer to /opt/lab/fixtures/ansible/inventory.sample.ini.
The changed=N on the PLAY RECAP line is the criterion for the verdict.
The handler name and the notify string must be exactly the same. Even a one-character difference is silently ignored.
Common mistake 1: putting a timestamp or random value in the template so that the rendered result differs every time. Then changed appears forever.
Common mistake 2: appending to a file with shell (>>) without a guard. Lines pile up on every run.

Create changes in the first run

Create /root/ans/idem/site.yml, run it, and save the output to /root/ans/idem/out/run1.txt. The changed of the first run must be 2 or more.

A few tasks that create directories, files, and configuration are enough. Save the entire run output to a file.

Make the second run give changed=0

Run the same playbook again and save it to /root/ans/idem/out/run2.txt. This time it must be changed=0.

Run the same playbook again. If changed is not 0, find in the log which task is the culprit.

Define a handler and call it with notify

Define a handler called reload app and call it with notify from the task that handles the configuration. When the handler runs, it must create /root/ans/idem/artifacts/reload.marker.

The handler name and the notify string must be exactly the same. Make the handler leave a marker file when it runs.

Keep the handler from running in the second run

The second run's log (run2.txt) must not contain RUNNING HANDLER. The first run's log must contain it.

notify fires only when the task is changed. The second run's log must not contain RUNNING HANDLER.

Put a creates guard on a shell command

Add one command/shell task and put a creates guard on it. That task writes one line to /root/ans/idem/artifacts/stamp.txt, and even if it runs twice, the file must contain only one line.

If you tell it the path of the file the command creates with creates, it is skipped from the second run on. Only one line must pile up in the log.

Set the reporting criteria yourself

Use changed_when at least once on a task that only queries, and failed_when at least once on a task that needs its own failure decision.

Set a command that only queries to changed_when: false, and use failed_when where you need your own failure decision.

Confirm no changes in check mode too

Run the playbook with --check and save the output to /root/ans/idem/out/check.txt. It must be changed=0 and failed=0.

If the state has already converged, a --check run must also give changed=0. If there is a module that does not support check mode, it fails.

Build the idempotency verdict report

Create /root/ans/idem/out/idempotency.json. It holds four keys: run1_changed (2 or more), run2_changed (0), handler_fired (1 or true), and verdict ("idempotent").

Summarize the changed counts of the two runs and the number of handler firings as JSON, and put in the verdict string.