Common Lessons Extracted from a Week of Incidents: Defaults, Green Lights, Human Responsibilities, and Locking in Fixes with Tests

Status: Published (2026-10-16). Compiled based solely on the facts verified in the previous five posts. The evidence and limitations for each part can be found in their respective posts.

Table of Contents

• 1. Invisible Labels and Defaults
• "Green Lights" and Actual State
• The Boundary Between Human Tasks and Automation
• Locking in Fixes with Tests
• Checklist for the Next Incident
• References

1. Invisible Labels and Defaults

Four out of the five incidents started from values that no one explicitly configured.

Their commonality is that they are not visible on the screen. Labels are not shown in the default node list output, update logs do not mention "do not reboot," policies look plausible at a glance, and regex patterns are short. Therefore, whenever you select something (advertised nodes, patterns, regexes), you need the habit of inspecting its labels and default rules first. kubectl get nodes --show-labels, commented-out defaults in configuration files, the "wildcards" section in product documentation, and regex boundary rules constitute that "inspection."

2. "Green Lights" and Actual State

In each incident, there was a signal saying "everything is fine." And those signals were not lying. They simply had a narrow field of view.

• Requests sent from within the cluster to the gateway truly succeeded. It was just that no one looked between the router and the advertised nodes.
• Nodes were genuinely Ready and auto-updates truly succeeded. It was just that no one checked the kernel modules in memory.
• unchanged was truly "nothing to change." It was just that the file read was outdated.
• The secret gate truly passed. It was just that no one looked at the folder where the values actually lived.
• The culprit finder truly didn't know the culprit. It was just that it didn't look at the .js path in the failure message.

During the same week, we saw something similar in the deployment tool's documentation. The ArgoCD documentation stated that CronJob tasks report a status of "Progressing" while running, but the built-in health check reported "Healthy" during that time. We raised an issue regarding the discrepancy between the documentation and reality, which was closed as completed on 2026-10-15 — argoproj/argo-cd#30209. If you do not know what a green light is observing, the green light tells you nothing.

Therefore, we decided to operate as follows: Verify from the output side. Verify whether deployment occurred not via pipeline green lights, but through operational files produced; whether policies changed not via apply output, but through values re-read from the cluster; and whether websites open from the outside, not the inside.

3. The Boundary Between Human Tasks and Automation

Most places where work stalled this week were boundaries.

Automation should not take over human responsibilities, but accurately handing off human responsibilities to humansis the job of automation. Hand-offs that proved effective this week took this shape: a single-line cause, copy-pasteable commands, a list of affected items, and a read command to verify whether it was done. Conversely, changes that proved ineffective were those that simply ended upon being committed to the repository—the DNS policy spent a month without anyone assigned to apply it to the cluster or commands to check if it had been applied. Handed-off tasks must be accompanied by verification commands and deadlines, and unverified hand-offs must remain visibly tracked as "Pending Application."

4. Locking in Fixes with Tests

The way to ensure fixes do not break again is through tests. However, this week also demonstrated that tests can also be wrong.

• Before introducing the credential gate, we first saw 13 places turn redon the pre-fix tree. This order is correct.
• The DNS policy test only checked the shapeof the policy string. Consequently, patterns that behaved contrary to intent (*.cluster.local) passed. Tests must assert meaning, such as "this name passes, that name is rejected."
• The trademark check was a protective test, but it mistakenly recognized 729 within a number as a trademark. Protective tests also require examples of "what must not be caught."

In summary, whenever adding a test, we verify three things: ① Is it red before the fix? ② Is it green after the fix? ③ Is it green on examples that must not be caught?

5. Checklist for the Next Incident

• Did you inspect labels, defaults, and wildcard rules before selecting a target?
• Did you verify "success" from the output side (externally, re-read cluster values, operational output files)?
• Does the file read by the apply command belong to the remote main branch, and is kubectl diff non-empty?
• On nodes with auto-updates enabled, can packages requiring reboots sneak in?
• If secrets entered the repository, were they rotated (not merely deleted from files)?
• Do tasks handed off to humans include commands, impact assessments, verification commands, and deadlines?
• Are new tests red before fixing, and green on examples that must not be caught?
• If the same failure occurs twice, does it halt and become visible to humans?

A Word from Titti: After stumbling five times, I realized we tripped over similar things every time: invisible defaults, narrow green lights, unclaimed tasks, and surface-level tests. Next time, let's open the checklist first before we start!

References

All materials were accessed on 2026-10-16.

• Documentation PRs and issues submitted this week: https://github.com/metallb/metallb/pull/3156, https://github.com/NVIDIA/cloud-native-docs/pull/524, https://github.com/argoproj/argo-cd/issues/30209
• The basis for each incident can be found in the "References" section of parts 1–5 of this series.