Implementing a Circuit Breaker and Verifying Its Transitions

Goal

Implement a circuit breaker yourself to see the transitions among CLOSED, OPEN, and HALF_OPEN with your own eyes, and get a feel, through an incident case, for why the decision of what to count as a failure is the most important.

Why it matters

A circuit breaker is not a device that fixes failures. It is a device that fails fast when a failure is certain, protecting the caller's connection pool from drying up. So it must be evaluated by isolation metrics, not performance metrics. The incidents a circuit creates in practice are mostly of two kinds. First, counting 4xx as failures and blocking even normal traffic — there is a case where the inventory service began returning 400 for a particular product, the circuit opened, and even lookups of normal products were blocked. Second, allowing only 1 HALF_OPEN trial call, so that it flaps between OPEN and HALF_OPEN. This lab assigns one step to each of those two traps, so you create them yourself and block them yourself.

Steps

1. Start /opt/app/flaky.py on 127.0.0.1:8110. GET /count returns the number of requests received so far as JSON.
2. Create /root/cb/breaker.py and call /fast 3 times. In /root/cb/state.json, write {"state":"CLOSED", ...}.
3. Call /always500 5 times to exceed the failure rate threshold. The settings are window 10, minimum calls 5, and failure rate 50%. In state.json, the state becomes OPEN.
4. Call 10 more times in the OPEN state. In /root/cb/fastfail.txt, write before=<n> after=<n> blocked=10 max_ms=<밀리초> (max_ms in milliseconds). before and after must be the same and max_ms must be under 50.
5. Set waitDurationInOpenState to 3 seconds, wait at least 3 seconds, and call once. In state.json, the state becomes HALF_OPEN.
6. In HALF_OPEN, make /fast succeed 3 times to return to CLOSED. The value permitted_in_half_open must be recorded as 3 or more.
7. With a new breaker, call /bad (400) 10 times. In /root/cb/ignore4xx.json, state must still be CLOSED and failures must be 0.
8. Run the whole scenario at once and leave three lines in /root/cb/transitions.log in this order: CLOSED->OPEN, OPEN->HALF_OPEN, and HALF_OPEN->CLOSED.

Notes

• Starting-point settings: window 10, minimum calls 5, failure rate 50%, slow call 3 seconds, OPEN wait 30 seconds (3 seconds in the lab), 3 HALF_OPEN trial calls
• Per-service thresholds: payment 30–40%, notifications 60–70%
• Common mistake 1: judging before minimumNumberOfCalls is filled, so that the circuit opens on a single failed call.
• Common mistake 2: making the fallback return None — the circuit has merely turned the outage into a different exception.

Prepare the downstream and a call counter

Start /opt/app/flaky.py on 127.0.0.1:8110. GET /count returns the number of requests received so far as JSON.

/opt/app/flaky.py tells you the number of requests it has received through /count. This value becomes the evidence in later steps.

Let calls through in the CLOSED state

Create /root/cb/breaker.py and call /fast 3 times. In /root/cb/state.json, write {"state":"CLOSED", ...}.

You must save the state to a file to be graded. Write the state name, the failure count, and the window size in JSON.

Make it OPEN with the failure rate threshold

Call /always500 5 times to exceed the failure rate threshold. The settings are window 10, minimum calls 5, and failure rate 50%. In state.json, the state becomes OPEN.

Do not judge before the minimum number of calls is filled. Implement window 10, minimum 5, and threshold 50% as given.

Prove that calls do not go to the downstream in OPEN

Call 10 more times in the OPEN state. In /root/cb/fastfail.txt, write before=<n> after=<n> blocked=10 max_ms=<밀리초> (max_ms in milliseconds). before and after must be the same and max_ms must be under 50.

The downstream's call counter must stay the same before and after blocking. And the response must be very fast.

Transition to HALF_OPEN after the wait time

Set waitDurationInOpenState to 3 seconds, wait at least 3 seconds, and call once. In state.json, the state becomes HALF_OPEN.

Record the time of entering OPEN and compare the elapsed time. The transition must be automatic so that no person has to intervene.

Return to CLOSED when trial calls succeed

In HALF_OPEN, make /fast succeed 3 times to return to CLOSED. The value permitted_in_half_open must be recorded as 3 or more.

If you set the number of trial calls to 1, flapping occurs. Implement it while thinking about why it must be 3 or more.

Do not count 4xx as failures

With a new breaker, call /bad (400) 10 times. In /root/cb/ignore4xx.json, state must still be CLOSED and failures must be 0.

A 400 is the client's fault, so it is not a matter for the circuit to step in. Manage what counts as a failure as a list.

Verify everything with the transition history

Run the whole scenario at once and leave three lines in /root/cb/transitions.log in this order: CLOSED->OPEN, OPEN->HALF_OPEN, and HALF_OPEN->CLOSED.

Join the earlier steps into one scenario so that the four state transitions are recorded in order.