Idempotent requests still need retry budgets

Goal

You will build an executor that keeps the status code, the safety of the request, the maximum count, and the overall time limit together.

Why it matters

A single success of a normal request does not guarantee boundary values or recovery from failure. In this lab you implement or test the contract of each function in small pieces and then connect it to a real run. It inspects results, exceptions, and stored state rather than just checking whether code exists or what the report says. Keep the code from the earlier steps as you move on to the next step.

Steps

In /root/work/idem-retry-budget-lab/service.py, make retryable(status) return True only for 429, 502, 503, and 504. All other statuses are False. This list is this lab's policy and is not the right answer for every service. Do the first setup with the following commands.

mkdir -p /root/work/idem-retry-budget-lab
cp /opt/fixtures/practice_depth/idem-retry-budget-lab/* /root/work/idem-retry-budget-lab/
cd /root/work/idem-retry-budget-lab

In /root/work/idem-retry-budget-lab/service.py, make replay_safe(method, key=None) return True only for GET, HEAD, PUT, DELETE (case-insensitive) or a POST that has a key that is not blank. PATCH is not allowed under this contract.
In /root/work/idem-retry-budget-lab/service.py, make remaining(deadline, now) return the larger of deadline-now and 0. The arguments are real numbers in seconds on a monotonic clock.
In /root/work/idem-retry-budget-lab/service.py, make backoff(attempt, base=0.5, cap=4.0) return min(cap, base*2**attempt). attempt must be 0–30, and base and cap must not be negative; a violation is a ValueError.
In /root/work/idem-retry-budget-lab/service.py, make jitter(delay, fraction) return delay*fraction. delay>=0 and 0<=fraction<=1 must hold; a violation is a ValueError.
In /root/work/idem-retry-budget-lab/service.py, make wait_delay(local, retry_after) return the larger of local and the Retry-After integer seconds. Unsupported values such as None, negative numbers, non-integers, and dates are interpreted as 0 seconds.
In /root/work/idem-retry-budget-lab/service.py, can_attempt(attempts, maximum, wait, budget) is True only when all of attempts < maximum, wait>=0, and budget>wait hold. attempts is the total number of calls already made.
In /root/work/idem-retry-budget-lab/service.py, implement execute(send, clock, sleep, method='GET', key=None, timeout=5.0, maximum=3, random_fraction=lambda:0.5). send(remaining seconds) returns {status: int, retry_after: optional value}. Fix the deadline at the start, retry according to the policy above, and return the last response if it cannot retry any more. timeout<=0 or maximum<1 is a ValueError, and running out of budget just before a call is a TimeoutError.

Notes

The packages are installed in the image. No internet connection or pip install is needed.
Do the setup copy only once, at the very beginning. If you copy again, your working files are reset.
Grading imports your service.py and runs it with different ids and a temporary store. Do not hard-code the results.
Grading for each step runs within 45 seconds. Do not create infinite loops or real waiting.

Narrow down the responses to retry

In /root/work/idem-retry-budget-lab/service.py, make retryable(status) return True only for 429, 502, 503, and 504. All other statuses are False. This list is this lab's policy and is not the right answer for every service. Do the first setup with the following commands.

mkdir -p /root/work/idem-retry-budget-lab
cp /opt/fixtures/practice_depth/idem-retry-budget-lab/* /root/work/idem-retry-budget-lab/
cd /root/work/idem-retry-budget-lab

Repeating a request that failed with an error like 400, 401, or 409 does not make the input, the permissions, or the conflict resolve themselves.

You can reproduce the grading directly with bash /opt/lab/checks/idem-retry-budget-lab/01-contract.sh. Save the file and run it again.

Decide whether it is fine to send the request again

In /root/work/idem-retry-budget-lab/service.py, make replay_safe(method, key=None) return True only for GET, HEAD, PUT, DELETE (case-insensitive) or a POST that has a key that is not blank. PATCH is not allowed under this contract.

Distinguish the premise that the server provides deduplication of the key. Merely putting a string in a header does not make it safe.

You can reproduce the grading directly with bash /opt/lab/checks/idem-retry-budget-lab/02-contract.sh. Save the file and run it again.

Share a single deadline

In /root/work/idem-retry-budget-lab/service.py, make remaining(deadline, now) return the larger of deadline-now and 0. The arguments are real numbers in seconds on a monotonic clock.

If you issue a new timeout for each attempt, the overall limit grows.

You can reproduce the grading directly with bash /opt/lab/checks/idem-retry-budget-lab/03-contract.sh. Save the file and run it again.

Put a cap on the backoff

In /root/work/idem-retry-budget-lab/service.py, make backoff(attempt, base=0.5, cap=4.0) return min(cap, base*2**attempt). attempt must be 0–30, and base and cap must not be negative; a violation is a ValueError.

attempt=0 is the wait after the first failure. Check the range so that the exponent calculation does not grow without bound.

You can reproduce the grading directly with bash /opt/lab/checks/idem-retry-budget-lab/04-contract.sh. Save the file and run it again.

Test jitter deterministically

In /root/work/idem-retry-budget-lab/service.py, make jitter(delay, fraction) return delay*fraction. delay>=0 and 0<=fraction<=1 must hold; a violation is a ValueError.

If you inject fraction instead of hiding the random number generation inside the function, you can reproduce both ends and the middle.

You can reproduce the grading directly with bash /opt/lab/checks/idem-retry-budget-lab/05-contract.sh. Save the file and run it again.

Do not retry earlier than the server's wait

In /root/work/idem-retry-budget-lab/service.py, make wait_delay(local, retry_after) return the larger of local and the Retry-After integer seconds. Unsupported values such as None, negative numbers, non-integers, and dates are interpreted as 0 seconds.

If the server asks for 7 seconds, it is 7 seconds even if the local backoff is 0.5 seconds. If the remaining budget is short, you give up in the next step.

You can reproduce the grading directly with bash /opt/lab/checks/idem-retry-budget-lab/06-contract.sh. Save the file and run it again.

Look at the attempt count and the wait budget together

In /root/work/idem-retry-budget-lab/service.py, can_attempt(attempts, maximum, wait, budget) is True only when all of attempts < maximum, wait>=0, and budget>wait hold. attempts is the total number of calls already made.

If the time remaining after the wait is exactly 0, you cannot start a new request.

You can reproduce the grading directly with bash /opt/lab/checks/idem-retry-budget-lab/07-contract.sh. Save the file and run it again.

Verify the whole executor with a fake clock

In /root/work/idem-retry-budget-lab/service.py, implement execute(send, clock, sleep, method='GET', key=None, timeout=5.0, maximum=3, random_fraction=lambda:0.5). send(remaining seconds) returns {status: int, retry_after: optional value}. Fix the deadline at the start, retry according to the policy above, and return the last response if it cannot retry any more. timeout<=0 or maximum<1 is a ValueError, and running out of budget just before a call is a TimeoutError.

The cost of the response is also included in the clock. The first failure is backoff(0). Check the time again after sleep as well, and reduce the timeout you give to send.

You can reproduce the grading directly with bash /opt/lab/checks/idem-retry-budget-lab/08-contract.sh. Save the file and run it again.