Contract tests for error-response regressions: design principles

Summary

You test known errors, unknown errors, private information, and the tracing header.

Why this matters

After an error message was changed, the retry logic of a mobile app broke. The test checked only that an exception occurred and did not compare the meaning of the status and the public body. A regression in which an internal exception is exposed to the user as it is slipped through the same gap.

How it works

Build a table of the normal codes and unknown, and compare the status and the public sentence. For the request id, check both sides of the length boundary. Check the key set and the Content-Type of the problem body, and observe on a real exception path whether the header and the body carry the same tracking value.

학생 테스트 → 정상 구현: 실제 시험 모두 통과
           └→ 계약 위반 구현: 해당 동작에서 실패
수집 실패·0개 실행·강제 종료 ≠ 결함 검출

A worksheet for reading the contract and predicting failures

What follows is not an answer key to memorize an implementation but a step-by-step code review. Each change fragment deliberately breaks the contract. Note that the normal case may still pass after the change. Before running it, predict which input, exception, or state you would observe to expose the difference, and after implementing it, compare that prediction with the result.

1. Keep a code on the business exception — test

Test the following public contract of the provided service.py: DomainError(code, message) is a subclass of Exception and stores the code in .code. str(exception) is the message. It must pass against the correct implementation and be caught, through a failure in the body of an actual test, in an implementation that breaks this contract. Keep the tests from the earlier steps and add a test_ function.

Basis for the judgment: Separate the code that a machine decides on from the internal message that a person sees. Do not modify the implementation file. Use pytest.raises to check the expected exception, and assert a concrete expected value for the normal result.

Faulty change fragment to review:

self.code = "invalid"

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

2. Map the code to a status — test

Test the following public contract of the provided service.py: status_for(code) is missing=404, conflict=409, invalid=422, and anything else=500. It must pass against the correct implementation and be caught, through a failure in the body of an actual test, in an implementation that breaks this contract. Keep the tests from the earlier steps and add a test_ function.

Basis for the judgment: Never treat an unknown code as a success. Do not modify the implementation file. Use pytest.raises to check the expected exception, and assert a concrete expected value for the normal result.

Faulty change fragment to review:

.get(code, 200)

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

3. Fix the public sentence — test

Test the following public contract of the provided service.py: public_message(code) is missing='Resource not found', conflict='State conflict', invalid='Invalid request', and anything else='Internal error'. It must pass against the correct implementation and be caught, through a failure in the body of an actual test, in an implementation that breaks this contract. Keep the tests from the earlier steps and add a test_ function.

Basis for the judgment: Even if the exception string contains a DB address or an internal path, it is not exposed. Do not modify the implementation file. Use pytest.raises to check the expected exception, and assert a concrete expected value for the normal result.

Faulty change fragment to review:

"invalid":"Internal error"

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

4. Restrict the request id — test

Test the following public contract of the provided service.py: request_id(value) returns the value as it is if it is a string of 1–32 characters made only of ASCII letters, digits, underscores, and hyphens, and 'untracked' otherwise. It must pass against the correct implementation and be caught, through a failure in the body of an actual test, in an implementation that breaks this contract. Keep the tests from the earlier steps and add a test_ function.

Basis for the judgment: Restrict both the length and the character set so that an arbitrary header is not reflected back. Do not modify the implementation file. Use pytest.raises to check the expected exception, and assert a concrete expected value for the normal result.

Faulty change fragment to review:

{1,64}

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

5. Build the problem body — test

Test the following public contract of the provided service.py: problem(code, rid) is a dictionary that has only type='urn:labhub:problem:'+code, title and detail=public_message(code), status=status_for(code), and request_id=request_id(rid). It must pass against the correct implementation and be caught, through a failure in the body of an actual test, in an implementation that breaks this contract. Keep the tests from the earlier steps and add a test_ function.

Basis for the judgment: If the body and the HTTP status disagree, the client cannot decide which value to trust. Do not modify the implementation file. Use pytest.raises to check the expected exception, and assert a concrete expected value for the normal result.

Faulty change fragment to review:

"status":500

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

6. Make the response format consistent — test

Test the following public contract of the provided service.py: response_for(code, rid) is a JSONResponse with problem as the body, status_for as the status, application/problem+json as the media_type, and X-Request-ID set to the normalized rid. It must pass against the correct implementation and be caught, through a failure in the body of an actual test, in an implementation that breaks this contract. Keep the tests from the earlier steps and add a test_ function.

Basis for the judgment: Returning a plain dictionary can turn even an error into a 200. Do not modify the implementation file. Use pytest.raises to check the expected exception, and assert a concrete expected value for the normal result.

Faulty change fragment to review:

media_type="application/json"

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

7. Connect the business exception handler — test

Test the following public contract of the provided service.py: install_handlers(app) registers a DomainError handler. It reads the X-Request-ID header and returns response_for for exc.code. It does not put the message of exc into the response. It must pass against the correct implementation and be caught, through a failure in the body of an actual test, in an implementation that breaks this contract. Keep the tests from the earlier steps and add a test_ function.

Basis for the judgment: Do not scatter the places where exceptions are caught; put them at the common boundary of the app. Do not modify the implementation file. Use pytest.raises to check the expected exception, and assert a concrete expected value for the normal result.

Faulty change fragment to review:

response_for("invalid", request.headers.get("x-request-id"))

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

8. Hide unexpected errors too — test

Test the following public contract of the provided service.py: create_app() installs the handlers and raises DomainError(code, internal message) at GET /fail/{code}. But when code=boom it raises RuntimeError. The RuntimeError handler returns a fixed 500 problem response with code=internal. It must pass against the correct implementation and be caught, through a failure in the body of an actual test, in an implementation that breaks this contract. Keep the tests from the earlier steps and add a test_ function.

Basis for the judgment: In the test, turn off re-raising of exceptions and check the bytes of the actual 500 response. Do not modify the implementation file. Use pytest.raises to check the expected exception, and assert a concrete expected value for the normal result.

Faulty change fragment to review:

response_for("invalid", request.headers.get("x-request-id"))

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

What it looks like in the field

The problem response in this lab is a teaching contract with type, title, status, detail, and request_id. It does not claim general internationalization or full standards conformance. A request id is a string used for tracing, not a means of authentication, and secret values must not be recorded as they are in production logs either. You may read the provided implementation, but grading uses a separate copy. Do not work around a defect by checking the wording of the source or by modifying files; check the execution results of the public interface.

What you will do in the next lab

Eight steps lead to one runnable result. Keep a code on the business exception — test → Map the code to a status — test → Fix the public sentence — test → Restrict the request id — test → Build the problem body — test → Make the response format consistent — test → Connect the business exception handler — test → Hide unexpected errors too — test.

Each step checks actual return values, exceptions, and state changes, not the fact that a function or file exists. After you see the answer, deliberately change a boundary comparison or the cleanup code and check which tests fail. Explain why the earlier tests are kept in the next steps, and write down one operating condition that this lab does not guarantee.