One order, two workflows: an event failure lab

Goal

Verify the boundaries of filters, permissions, and duplicate handling with real events, Workflows, and an order ledger.

Why it matters

An HTTP 200 is not business completion. This VM prepares a real k3s, Argo Events 1.9.11, Workflows 4.1.3, and a local SQLite ledger. The student fixes the field of a wrong Sensor and compares the actual runs and business effects of the control inputs the helper sends. Each step uses a separate Sensor and port, so steps 2–8 can be started even if you skip the earlier steps. The preparation for step 9 only fills in materials of earlier steps you did not touch and does not overwrite the student's fixes or report.

Steps

With kubectl, read the EventSource lesson, the EventBus default, and the Role sensor-submit. Record namespace=argo-events, eventsource=lesson, eventbus=default, submit_account=sensor-submit, workflow_account=workflow-runner, and http_is_completion=false in /root/capa-events/inventory.json. HTTP intake and Workflow completion are different boundaries.
In /root/capa-events/path.yaml, change only the path of the first data filter from body.event.kind to body.kind and run the helper with run path. The helper sends flat and nested bodies to the wrong baseline and to the fixed version. The baseline must create a real Workflow only for nested, and the fixed version for both.
In /root/capa-events/regex.yaml, change only the value of the first data filter to ["^order[.]created$"] and run regex. pre-orderXcreated-tail is approved in the baseline but rejected after the fix, and the order.created control group must continue to run.
Keep the existing data filter in /root/capa-events/types.yaml and add return type(event.body.enabled) == "boolean" and event.body.enabled == true to filters.script. With run types, compare false, the string true, the number 1, a missing field, and the boolean true. After the fix, only the last input must run.
In /root/capa-events/logic.yaml, change only filters.dataLogicalOperator from or to and and run logic. order.deleted with enabled=true must be rejected after the fix, and order.created must run. Keep the two filters and the other settings.
Change roleRef.name in /root/capa-events/repair-binding.yaml to sensor-submit. Keep the name sensor-repair, namespace argo-events, kind Role, apiGroup rbac.authorization.k8s.io, and a single subject, the argo-events ServiceAccount sensor-repair. Leave rbac.yaml as is and run rbac. The Role keeps only Workflow create and you do not make an additional binding. The helper preserves the actual forbidden of the unprivileged baseline and checks a new request from the fixed account.
Keep /root/capa-events/duplicate.yaml and run duplicate. From two identical HTTP bodies, confirm two different arrival labels and two Workflow UIDs. The order-id of each Workflow must be the same business key, and the actual Pods must exit successfully. This experiment is two HTTP requests and is not a broker redelivery experiment.
In the last URL of the Workflow container args in /root/capa-events/ledger.yaml, change only /naive to /safe and run ledger. Keep the address, port, the amount 1250, and the passing of the business key. The baseline must have 2 ledger effects and the fixed version 1 effect. Both Workflows succeed and the fixed version's results are applied and duplicate. The helper's request with the same key and a different amount must be a 409.
After completing the earlier experiments, run boundaries. In the ledger API, compare 16 concurrent requests, a request whose response was not read, a repeat request with the same key after a ledger service restart, and 5 kinds of wrong amounts. In /root/capa-events/report.json, write the current run ID for duplicate_attempt and boundary_attempt, and for workflow_uids the two actual UIDs of duplicate in ascending order. Record http_requests=2, workflow_executions=2, safe_effects=1, concurrent_requests=16, restart_preserved=true, broker_redelivery_proven=false, and external_exactly_once_proven=false. A ledger restart is not a VM or broker restart.

Notes

Run: python3 /opt/fixtures/capa-events/runtime.py run path (use the relevant case in place of path).
Status and re-observation: status path and wait path of the same helper. A wait expiring is not grounds for failure or resending.
A finished identical submission is reused. Use run path --new-attempt only when you truly need a fresh comparison.
/var/lib/labhub/capa-events/runs/<attempt>/result.json that the helper points to is the original observation. Do not edit it directly; read the Workflow metadata.uid and labels.arrival in after.events and after.executions.
After reading the result, compare it with the current API using kubectl -n argo-events get workflows -l lesson=capa-events -o yaml.
The lab is 55 minutes. If it runs long, extend it with the +time button before it ends. After the session ends, files and the ledger are not kept.
The concurrent request and unconfirmed response experiments are ledger API experiments. It is a single broker and a single VM, and it does not prove high availability, broker redelivery, or exactly-once for external payments and emails.
Official filter documentation · Service accounts · SQLite transactions

Find the boundary between intake and business completion

With kubectl, read the EventSource lesson, the EventBus default, and the Role sensor-submit. Record namespace=argo-events, eventsource=lesson, eventbus=default, submit_account=sensor-submit, workflow_account=workflow-runner, and http_is_completion=false in /root/capa-events/inventory.json. HTTP intake and Workflow completion are different boundaries.

Read the actual objects with kubectl -n argo-events get eventsource lesson -o yaml and get role sensor-submit -o yaml.

Distinguish a nonexistent JSON path using the normal control group

In /root/capa-events/path.yaml, change only the path of the first data filter from body.event.kind to body.kind and run the helper with run path. The helper sends flat and nested bodies to the wrong baseline and to the fixed version. The baseline must create a real Workflow only for nested, and the fixed version for both.

Distinguish the HTTP body from the body of the Argo event envelope. Do not look only at there being no run; also confirm the success of the nested control group.

Prevent regular expression over-matching

In /root/capa-events/regex.yaml, change only the value of the first data filter to ["^order[.]created$"] and run regex. pre-orderXcreated-tail is approved in the baseline but rejected after the fix, and the order.created control group must continue to run.

A dot is by default any character, and without anchors a substring can match too.

Distinguish a truthy-looking value from a real JSON boolean

Keep the existing data filter in /root/capa-events/types.yaml and add return type(event.body.enabled) == "boolean" and event.body.enabled == true to filters.script. With run types, compare false, the string true, the number 1, a missing field, and the boolean true. After the fix, only the last input must run.

Do not assume the original type from only the converted result of type: bool. Use Lua's type together with a value comparison.

Reject a request where only one condition is true

In /root/capa-events/logic.yaml, change only filters.dataLogicalOperator from or to and and run logic. order.deleted with enabled=true must be rejected after the fix, and order.created must run. Keep the two filters and the other settings.

Even if the meaning of each condition is right, if the operator that joins the two conditions differs, the allowed set differs.

Recover a real forbidden with minimal permissions

Change roleRef.name in /root/capa-events/repair-binding.yaml to sensor-submit. Keep the name sensor-repair, namespace argo-events, kind Role, apiGroup rbac.authorization.k8s.io, and a single subject, the argo-events ServiceAccount sensor-repair. Leave rbac.yaml as is and run rbac. The Role keeps only Workflow create and you do not make an additional binding. The helper preserves the actual forbidden of the unprivileged baseline and checks a new request from the fixed account.

The Sensor account that creates the Workflow and the execution account of the created Workflow are different. Attach the existing minimal Role, not an administrator.

Trace two different executions of the same order

Keep /root/capa-events/duplicate.yaml and run duplicate. From two identical HTTP bodies, confirm two different arrival labels and two Workflow UIDs. The order-id of each Workflow must be the same business key, and the actual Pods must exit successfully. This experiment is two HTTP requests and is not a broker redelivery experiment.

In the helper's result path, compare events and after.executions. Even if the business key is the same, the arrival ID and the execution UID can differ.

Limit two executions to one business effect

In the last URL of the Workflow container args in /root/capa-events/ledger.yaml, change only /naive to /safe and run ledger. Keep the address, port, the amount 1250, and the passing of the business key. The baseline must have 2 ledger effects and the fixed version 1 effect. Both Workflows succeed and the fixed version's results are applied and duplicate. The helper's request with the same key and a different amount must be a 409.

The URL is in spec.triggers[0].template.k8s.source.resource.spec.templates[0].container.args. Read the original address before saving.

Report the concurrent request and restart boundaries and the unverified scope

After completing the earlier experiments, run boundaries. In the ledger API, compare 16 concurrent requests, a request whose response was not read, a repeat request with the same key after a ledger service restart, and 5 kinds of wrong amounts. In /root/capa-events/report.json, write the current run ID for duplicate_attempt and boundary_attempt, and for workflow_uids the two actual UIDs of duplicate in ascending order. Record http_requests=2, workflow_executions=2, safe_effects=1, concurrent_requests=16, restart_preserved=true, broker_redelivery_proven=false, and external_exactly_once_proven=false. A ledger restart is not a VM or broker restart.

Read the original result.json that runtime.py status CASE points to. If the observation wait has not ended, do not create a new experiment; use wait on the same handle.