The reconcile ran twice - lease expiry, takeover, and incident diagnosis

Goal

Work with the five fields of a Lease yourself to create renewal, expiry, and takeover, set the minimum permissions needed for leader election, prove with a field manager conflict what actually happens when there are two leaders, and bundle it into an operations checklist.

Why it matters

The reason to run two or more Operators is availability. But the reconcile loop is code that changes cluster state, so two must not run at the same time. So Kubernetes writes "who is the leader right now" on one very small object, and the leader renews periodically to announce that it is alive. What matters here is that this object has no field like "expired." Whether it is alive is decided by a calculation comparing renewTime + leaseDurationSeconds with now, and that calculation is done by each instance with its own clock. If clocks drift or the API server responds slowly, an interval arises in which two instances both believe they are the leader. What happens when two reconcile loops try to create the same resource in that interval, and where to find the traces, is the subject of this lab.

Steps

1. Create the namespace op-leader and write a coordination.k8s.io/v1 Lease widget-operator in /root/op-leader/lease-widget.yaml — holderIdentity is widget-operator-0, leaseDurationSeconds is 15, acquireTime and renewTime are the current time, and leaseTransitions is 0. After applying, save the output of kubectl -n op-leader get lease widget-operator -o yaml to /root/op-leader/lease-initial.yaml.
2. Do by hand three times what a leader does — run a patch that updates only renewTime to the current time three times at 1-second intervals, and each time leave the updated renewTime in /root/op-leader/renew-log.txt, one line each (three lines). Do not touch holderIdentity or leaseTransitions.
3. Create two more leases — /root/op-leader/lease-stale.yaml is stale-operator (holder stale-operator-0, leaseDurationSeconds 15, renewTime set to 10 minutes ago), and /root/op-leader/lease-fresh.yaml is fresh-operator (holder fresh-operator-0, leaseDurationSeconds 86400, renewTime now). Write two lines in /root/op-leader/leases.tsv in the form <임차이름><탭><기대: EXPIRED 또는 LIVE> (lease name, a tab, and the expectation, either EXPIRED or LIVE), and make /root/op-leader/expiry.sh classify each lease by comparing its renewTime + leaseDurationSeconds with now, print OK … if it matches and MISMATCH … if it does not, to standard output only, and exit with a nonzero code if even one line is wrong. Save the output to /root/op-leader/expiry.txt.
4. Treat the widget-operator lease as having been taken by another instance and change four values with a single patch — holderIdentity to widget-operator-1, acquireTime and renewTime to the current time, and leaseTransitions to 1. Leave four lines in /root/op-leader/takeover.txt — before-holder=, after-holder=, before-transitions=, and after-transitions=.
5. Try to renew with a seconds-resolution timestamp — patch renewTime with a value that has no decimal point, such as 2026-01-01T00:00:00Z. Collect the result in /root/op-leader/microtime-error.txt — the first line is patch-rc=<종료 코드> (the exit code), and below it you paste the sentence the server produced exactly as it was.
6. Put three objects in /root/op-leader/rbac.yaml and apply it — a ServiceAccount widget-operator, a Role leader-election (only get, create, and update on leases in the coordination.k8s.io group), and a RoleBinding leader-election that connects the two. Then run kubectl auth can-i <동사> leases.coordination.k8s.io --as=system:serviceaccount:op-leader:widget-operator -n op-leader four times, for get, create, update, and delete (replacing the placeholder with the verb), and save the results to /root/op-leader/rbac-check.txt as four lines in the form <동사>=<yes 또는 no> (the verb, then yes or no).
7. Reproduce a situation where two instances both believe they are the leader — write a ConfigMap order-1 (namespace op-leader, data.phase of A) in /root/op-leader/child-a.yaml and apply it with kubectl apply --server-side --field-manager=widget-operator-0. Then write a ConfigMap with the same name in /root/op-leader/child-b.yaml with data.phase of B and try applying it with --field-manager=widget-operator-1. Collect the result in /root/op-leader/split-brain.txt — apply-b-rc=<종료 코드> (the exit code), the sentence the server produced, and on the last line final-phase=<지금 값> (the current value).
8. Create /root/op-leader/lease-audit.sh — print every lease in op-leader, sorted, one line each in the form <이름> holder=<홀더> transitions=<전환 횟수> duration=<임차 길이> (name, holder, transition count, and lease duration), to standard output only (do not include age or timestamps). Save the output to /root/op-leader/lease-audit.txt, and write the operating rules in /root/op-leader/runbook.txt — the names of all three values --leader-elect-lease-duration, --leader-elect-renew-deadline, and --leader-elect-retry-period must appear, and it must include why the renew deadline must be shorter than the lease duration and what to suspect when the transition count keeps growing.

Notes

• acquireTime and renewTime are MicroTime, so they need six digits after the decimal point.
• A renewal changes only renewTime, while a takeover changes the holder, acquireTime, and leaseTransitions together.
• The verbs needed for leader election are only three: get, create, and update.
• With kubectl auth can-i --as= you can ask about another subject's permissions on its behalf.
• Common mistake: raising leaseTransitions along with every renewal, making the transition count meaningless.
• Common mistake: trying to find whether a lease has expired in some field of the object — you have to compute it.
• Reference: https://kubernetes.io/docs/concepts/architecture/leases/
• Reference: https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/lease-v1/
• Reference: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

One lease decides the leader

Create the namespace op-leader and write a coordination.k8s.io/v1 Lease widget-operator in /root/op-leader/lease-widget.yaml — holderIdentity is widget-operator-0, leaseDurationSeconds is 15, acquireTime and renewTime are the current time, and leaseTransitions is 0. After applying, save the output of kubectl -n op-leader get lease widget-operator -o yaml to /root/op-leader/lease-initial.yaml.

The two time fields are MicroTime, not an ordinary Time, so they need six digits after the decimal point (for example, 2026-09-17T13:37:24.807518Z). With GNU date, you can produce it directly with date -u +%Y-%m-%dT%H:%M:%S.%6NZ. leaseTransitions is "the number of times the leader has changed," so it starts at 0.

The leader renews without rest

Do by hand three times what a leader does — run a patch that updates only renewTime to the current time three times at 1-second intervals, and each time leave the updated renewTime in /root/op-leader/renew-log.txt, one line each (three lines). Do not touch holderIdentity or leaseTransitions.

A renewal is only a signal that "I am still alive," so neither the holder nor the transition count changes. If the transition count rises on every renewal, it means the leader is changing every time, which is a state worth setting an alert on in operations. Check that the three lines grow in chronological order.

Whether it is alive is decided by calculation

Create two more leases — /root/op-leader/lease-stale.yaml is stale-operator (holder stale-operator-0, leaseDurationSeconds 15, renewTime set to 10 minutes ago), and /root/op-leader/lease-fresh.yaml is fresh-operator (holder fresh-operator-0, leaseDurationSeconds 86400, renewTime now). Write two lines in /root/op-leader/leases.tsv in the form <임차이름><탭><기대: EXPIRED 또는 LIVE> (lease name, a tab, and the expectation, either EXPIRED or LIVE), and make /root/op-leader/expiry.sh classify each lease by comparing its renewTime + leaseDurationSeconds with now, print OK … if it matches and MISMATCH … if it does not, to standard output only, and exit with a nonzero code if even one line is wrong. Save the output to /root/op-leader/expiry.txt.

A lease object has no field like "expired." Each candidate just computes it with its own clock — so if clocks drift, the judgment splits even when looking at the same object. For the calculation, convert to seconds with date -u -d "<시각>" +%s (replacing the placeholder with the timestamp) and add.

A takeover shows up in the transition count

Treat the widget-operator lease as having been taken by another instance and change four values with a single patch — holderIdentity to widget-operator-1, acquireTime and renewTime to the current time, and leaseTransitions to 1. Leave four lines in /root/op-leader/takeover.txt — before-holder=, after-holder=, before-transitions=, and after-transitions=.

What separates a takeover from a renewal is acquireTime and leaseTransitions. A renewal moves only renewTime, but in a takeover the holder changes, so you write anew when it was acquired and raise the transition count by one. If you watch this number, you can see how often the leader changes.

One time format blocks the renewal

Try to renew with a seconds-resolution timestamp — patch renewTime with a value that has no decimal point, such as 2026-01-01T00:00:00Z. Collect the result in /root/op-leader/microtime-error.txt — the first line is patch-rc=<종료 코드> (the exit code), and below it you paste the sentence the server produced exactly as it was.

This field is MicroTime, not Time. If the format differs, it is blocked at parsing, not at the value, and the error sentence states the expected format as it is. If you write a controller yourself and use the standard library's default format as it is, you get blocked right here and the renewal fails entirely.

The minimum permissions needed for leader election

Put three objects in /root/op-leader/rbac.yaml and apply it — a ServiceAccount widget-operator, a Role leader-election (only get, create, and update on leases in the coordination.k8s.io group), and a RoleBinding leader-election that connects the two. Then run kubectl auth can-i <동사> leases.coordination.k8s.io --as=system:serviceaccount:op-leader:widget-operator -n op-leader four times, for get, create, update, and delete (replacing the placeholder with the verb), and save the results to /root/op-leader/rbac-check.txt as four lines in the form <동사>=<yes 또는 no> (the verb, then yes or no).

Leader election needs only three verbs — read, create if absent, and renew periodically. There is nothing to delete. You do not need watch either, because candidates check expiry not by watching but by periodic queries. If you grant broad permissions, a path opens for accidentally deleting someone else's lease.

With two leaders, they fight over the same field

Reproduce a situation where two instances both believe they are the leader — write a ConfigMap order-1 (namespace op-leader, data.phase of A) in /root/op-leader/child-a.yaml and apply it with kubectl apply --server-side --field-manager=widget-operator-0. Then write a ConfigMap with the same name in /root/op-leader/child-b.yaml with data.phase of B and try applying it with --field-manager=widget-operator-1. Collect the result in /root/op-leader/split-brain.txt — apply-b-rc=<종료 코드> (the exit code), the sentence the server produced, and on the last line final-phase=<지금 값> (the current value).

Server-side apply records "who manages this value" for each field. If another manager tries to write the same field with a different value, the API server does not quietly overwrite it but reports a conflict. This mechanism makes visible what happens when leader election breaks down and two reconcile loops erase each other's results.

Bundle it into a checklist and operating rules

Create /root/op-leader/lease-audit.sh — print every lease in op-leader, sorted, one line each in the form <이름> holder=<홀더> transitions=<전환 횟수> duration=<임차 길이> (name, holder, transition count, and lease duration), to standard output only (do not include age or timestamps). Save the output to /root/op-leader/lease-audit.txt, and write the operating rules in /root/op-leader/runbook.txt — the names of all three values --leader-elect-lease-duration, --leader-elect-renew-deadline, and --leader-elect-retry-period must appear, and it must include why the renew deadline must be shorter than the lease duration and what to suspect when the transition count keeps growing.

A checklist is a condensed version of "what looks abnormal when you see it." If the holder stays the same but only the transition count grows, it means takeovers are repeating, and the cause is usually clock skew or API server latency. Explain the relationship among the three values based on the official documentation's defaults (15 seconds, 10 seconds, 2 seconds).