The response failed, but the business operation committed

Goal

You compare real Istio retries with business records, and implement a transaction boundary that prevents duplicates of the same business operation even when a response timeout or a process kill occurs.

Why it matters

A 200 is not proof of a business operation happening once, and a 504 is not proof of a rollback. If you create a new business key for each send attempt of the same order, the duplicate-prevention device is also neutralized. Conversely, merging different orders with the same content is also an error. This lab covers a synthetic HTTP business operation on a real mesh together with a separate transaction-code fix. Your code fix applies to the temporary SQLite test and is not a feature that automatically replaces the HTTP server code.

Steps

1. Save the entire output of python3 /opt/fixtures/ica-retry/runtime.py inventory to /root/ica-retry/inventory.json. The namespace is ica-retry, and the real proxies of client and upstream must be ready.
2. In /root/ica-retry/routes.yaml, create a networking.istio.io/v1 VirtualService retry-lab. The namespace is ica-retry, hosts and all destinations are upstream.ica-retry.svc.cluster.local, and the port is 8080. The first route, flaky, has two exact matches /flaky and /naive, timeout 1s, attempts 2, perTryTimeout 1s, retryOn '503', and retryIgnorePreviousHosts false. The second, lost, has exact /lost, timeout 500ms, and attempts 0. The last, control, has no match, the same destination, and attempts 0. After applying, run collect retry and compare the actual receipt and business counts in retry.json.
3. Run collect timeout of the same tool. timeout.json must contain together the client's 504 for /lost, the server's committed response, and the retrieval result of calling /charge with the same key. Do not make up and write numbers yourself.
4. Save conflict.json with collect conflict. Check that a request that changes the amount from 100 to 200 under the same business key gets a 409 and the existing business amount remains 100.
5. Save durability.json with collect durability. The tool actually deletes and recreates only the upstream Pod inside this VM. The client UID must be the same, and the upstream UID and server instance must differ. The same business key and response ID must be preserved.
6. Copy /opt/fixtures/ica-retry/broken.py to /root/ica-retry/transaction.py and fix the split-commit defect in charge's default execution. Keep initialize, charge, Conflict, and the forced-kill points after-business, after-key, and after-commit. Before the commit, both the business operation and the key must be 0, after the commit both must remain as 1, and after a retry the business operation must always be 1.
7. Check 8 concurrent requests with the same key using transaction_probe.py /root/ica-retry/transaction.py identity. There must be only one new processing, a change of customer or amount under the same key must be a Conflict, a change in JSON order must give the same response, and the same content under a different key must be a separate business operation. Reject a wrong amount and an unknown field with ValueError.
8. In /root/ica-retry/incident.json, record additional attempts 2, maximum attempts 3, naive business operations 3, and duplicate-prevented business operations 1. timeout_means_rollback and external_payment_exactly_once are false. Write changed_payload_status as the actual conflict code, durable_scope as same-vm-disk, and atomic_scope as single-sqlite-file. Leave the earlier real observations as they are for final grading.

Notes

• Example command: python3 /opt/fixtures/ica-retry/runtime.py collect timeout. If an observation file exists, the existing result is preserved. If you need a new test, keep your own observation file under a different name and then run it.
• A policy applied for the first time needs time to propagate to Envoy. If collect reports a route mismatch, check the actual proxy-config routes.
• broken.py is a deliberately wrong answer that reproduces the split-commit incident. Grading does not fix the student file and runs it on a temporary DB.
• Do not memorize the success ratio or response speed of a load experiment as the answer. Look at the state per business key.
• It covers only preservation on the disk of the same VM. It does not guarantee VM termination, a power cut, or exactly-once for external payments. Do not put in personal data or real payment information.

Recording the real proxies and Pod identities

Record the UIDs of the two Pods with the inventory command. Do not look only at the Running names; check that the real proxies are ready.

Telling three receipts from three business operations

Apply routes.yaml and run collect retry. attempts is additional attempts, and look separately at the business ledgers of /naive and /flaky.

Retrieving a business operation already committed after a 504

Check the 500ms budget and the no-retry condition of /lost. collect timeout asks /charge again with the same business key.

Rejecting a different amount under the same key

The two requests of collect conflict have the same business key and differ only in the amount. Check that the existing business operation and response were preserved.

The same business response even after replacing the Pod

collect durability replaces only this VM's upstream. The Pod UID and server instance must change and the business ID must be the same.

Fixing the split-commit defect in code

Copy broken.py to transaction.py and read it. Find whether a commit is sandwiched between the business write and the duplicate record in charge's default execution.

The boundary between concurrent requests and separate business operations

Merge concurrent requests with the same key into one, and keep identical content under a different key as a separate business operation. Also check the rejection of content changes and the JSON order.

An incident report that separates the response from the business operation

Distinguish the maximum number of attempts from the actual business effect, and limit the scope of the guarantee to a single SQLite file and the disk of the same VM.