Export JSON Lines without collecting every row: design principles

Summary

You connect lazy generation, line boundaries, public fields, and an output cap to an HTTP stream.

Why this matters

When an administrator downloaded all the orders, the server's memory spiked. The export function had gathered every row into a list before building the JSON. It was changed to line-by-line output, but newlines inside the content broke the real line boundaries and the internal cost went out unchanged. Streaming is not a matter of changing one return type; it means deciding lazy evaluation and the representation contract together.

How it works

Validate each row, project only the public fields, encode it as JSON, and append one newline to the end of the line. The generator does not consume the input just by being called, and taking one item advances only one row. The output cap is validated as an integer that is not a bool. At the end you connect it to a FastAPI StreamingResponse and parse the Content-Type and the downloaded lines again.

입력 iterator → 한 행 검증 → 공개 투영 → JSON + 개행 → StreamingResponse

A worksheet for reading the contract and predicting failures

What follows is not an answer key to memorize an implementation but a step-by-step code review. Each change fragment deliberately breaks the contract. Note that the normal case may still pass after the change. Before running it, predict which input, exception, or state you would observe to expose the difference, and after implementing it, compare that prediction with the result.

1. Validate the row contract

validate_row(row) returns row when it is a dict whose id is a positive int (excluding bool) and whose name is a non-empty str. Everything else is ValueError. Extra internal fields are allowed.

Basis for the judgment: distinguish bool from numbers and treat an empty name as an error.

Faulty change fragment to review:

not isinstance(row.get("id"), int)

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

2. Build only public rows

project(row) returns a new dict that has only id and name after validate_row. The internal fields of the original are preserved as they are.

Basis for the judgment: the export path must apply the same public-field policy as an ordinary API.

Faulty change fragment to review:

"name":row["name"], "internal_cost":row.get("internal_cost")}

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

3. Encode while preserving line boundaries

encode_line(row) is a str that JSON-encodes the result of project with ensure_ascii=False, separators=(',',':'), and sort_keys=True, and appends exactly one trailing newline character at the end. A newline inside name must be a JSON escape.

Basis for the judgment: if you build JSON by appending strings, the format breaks at quotes and newlines.

Faulty change fragment to review:

ensure_ascii=True

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

4. Validate the output count cap

validate_max(value) returns the value as it is only for an int from 1 to 1000 that is not a bool, and everything else is ValueError.

Basis for the judgment: require a cap from the caller so that an infinite input is not read to the end by mistake.

Faulty change fragment to review:

<= 1001

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

5. Consume only the rows you need

take_rows(rows, maximum) is an iterator that lazily returns at most maximum rows, using islice or similar. It validates maximum when called, and each next consumes the input only once.

Basis for the judgment: the moment you turn it into list(rows), you can no longer handle infinite input or very large input.

Faulty change fragment to review:

islice(list(rows), validate_max(maximum))

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

6. Serialize rows lazily

json_lines(rows, maximum=100) yields encode_line for each row received from take_rows. It does not return a string or a list with everything combined.

Basis for the judgment: keep object selection and representation conversion each as a lazy stage.

Faulty change fragment to review:

for row in list(take_rows(rows, maximum)):

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

7. Validate the downloaded lines again

decode_lines(text) runs json.loads and then validate_row on each non-empty line of splitlines and returns a list. An empty string gives [], and an empty line in the middle is ValueError.

Basis for the judgment: distinguish an empty file from a malformed empty record.

Faulty change fragment to review:

continue

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

8. Complete the HTTP download

create_app(rows) returns json_lines(rows, 100) as an application/x-ndjson StreamingResponse at GET /export. rows is a list that can be iterated again. There must be no internal fields, and the content and order of each row must be preserved.

Basis for the judgment: check, together with a generator test, that the code does not just state streaming in the Content-Type while gathering everything internally.

Faulty change fragment to review:

media_type="application/json"

Compare it with the public contract of the function this fragment sits in. If a single success case does not tell them apart, choose as your observation target an input that should be rejected or the state after a failure.

What it looks like in the field

TestClient buffers the response, so it does not prove the delay of the first byte over the network or the whole memory cap. Whether evaluation is lazy is checked with a separate counting generator. Once the stream has started, it is hard to change the status to a normal error JSON when you meet a bad row. A real service has to decide which to pick among prior validation, a per-row error format, and an abort policy.

What you will do in the next lab

Eight steps lead to one runnable result. Validate the row contract → build only public rows → encode while preserving line boundaries → validate the output count cap → consume only the rows you need → serialize rows lazily → validate the downloaded lines again → complete the HTTP download.

Each step checks actual return values, exceptions, and state changes, not the fact that a function or file exists. After you see the answer, deliberately change a boundary comparison or the cleanup code and check which tests fail. Explain why the earlier tests are kept in the next steps, and write down one operating condition that this lab does not guarantee.