Export JSON Lines without collecting every row

Goal

You connect lazy generation, line boundaries, public fields, and an output cap to an HTTP stream.

Why it matters

When an administrator downloaded all the orders, the server's memory spiked. The export function had gathered every row into a list before building the JSON. It was changed to line-by-line output, but newlines inside the content broke the real line boundaries and the internal cost went out unchanged. Streaming is not a matter of changing one return type; it means deciding lazy evaluation and the representation contract together.

Steps

1. In /root/work/fa-jsonl-export-lab/service.py, validate_row(row) returns row when it is a dict whose id is a positive int (excluding bool) and whose name is a non-empty str. Everything else is ValueError. Extra internal fields are allowed.

Prepare this once at the start. Existing files are not overwritten.

mkdir -p /root/work/fa-jsonl-export-lab
test -e /root/work/fa-jsonl-export-lab/service.py || cp /opt/fixtures/ten_labs/fa-jsonl-export-lab/service.py /root/work/fa-jsonl-export-lab/service.py
cd /root/work/fa-jsonl-export-lab

1. In /root/work/fa-jsonl-export-lab/service.py, project(row) returns a new dict that has only id and name after validate_row. The internal fields of the original are preserved as they are.

2. In /root/work/fa-jsonl-export-lab/service.py, encode_line(row) is a str that JSON-encodes the result of project with ensure_ascii=False, separators=(',',':'), and sort_keys=True, and appends exactly one trailing newline character at the end. A newline inside name must be a JSON escape.

3. In /root/work/fa-jsonl-export-lab/service.py, validate_max(value) returns the value as it is only for an int from 1 to 1000 that is not a bool, and everything else is ValueError.

4. In /root/work/fa-jsonl-export-lab/service.py, take_rows(rows, maximum) is an iterator that lazily returns at most maximum rows, using islice or similar. It validates maximum when called, and each next consumes the input only once.

5. In /root/work/fa-jsonl-export-lab/service.py, json_lines(rows, maximum=100) yields encode_line for each row received from take_rows. It does not return a string or a list with everything combined.

6. In /root/work/fa-jsonl-export-lab/service.py, decode_lines(text) runs json.loads and then validate_row on each non-empty line of splitlines and returns a list. An empty string gives [], and an empty line in the middle is ValueError.

7. In /root/work/fa-jsonl-export-lab/service.py, create_app(rows) returns json_lines(rows, 100) as an application/x-ndjson StreamingResponse at GET /export. rows is a list that can be iterated again. There must be no internal fields, and the content and order of each row must be preserved.

Notes

• You work in the existing lab-dev environment with no internet and no package installation.
• Each step runs within a 45-second grading budget. Do not add real sleeps or network calls.
• The grader loads the submitted module fresh and checks it with independent inputs and a temporary DB. Implement the contract instead of returning the expected values as constants.
• FastAPI official documentation · pytest official documentation · Python sqlite3
• Limitation: TestClient buffers the response, so it does not prove the delay of the first byte over the network or the whole memory cap. Whether evaluation is lazy is checked with a separate counting generator. Once the stream has started, it is hard to change the status to a normal error JSON when you meet a bad row. A real service has to decide which to pick among prior validation, a per-row error format, and an abort policy.

Validate the row contract

In /root/work/fa-jsonl-export-lab/service.py, validate_row(row) returns row when it is a dict whose id is a positive int (excluding bool) and whose name is a non-empty str. Everything else is ValueError. Extra internal fields are allowed.

Prepare this once at the start. Existing files are not overwritten.

mkdir -p /root/work/fa-jsonl-export-lab
test -e /root/work/fa-jsonl-export-lab/service.py || cp /opt/fixtures/ten_labs/fa-jsonl-export-lab/service.py /root/work/fa-jsonl-export-lab/service.py
cd /root/work/fa-jsonl-export-lab

Distinguish bool from numbers and treat an empty name as an error.

After saving, check with bash /opt/lab/checks/fa-jsonl-export-lab/01-contract.sh.

Build only public rows

In /root/work/fa-jsonl-export-lab/service.py, project(row) returns a new dict that has only id and name after validate_row. The internal fields of the original are preserved as they are.

The export path must apply the same public-field policy as an ordinary API.

After saving, check with bash /opt/lab/checks/fa-jsonl-export-lab/02-contract.sh.

Encode while preserving line boundaries

In /root/work/fa-jsonl-export-lab/service.py, encode_line(row) is a str that JSON-encodes the result of project with ensure_ascii=False, separators=(',',':'), and sort_keys=True, and appends exactly one trailing newline character at the end. A newline inside name must be a JSON escape.

If you build JSON by appending strings, the format breaks at quotes and newlines.

After saving, check with bash /opt/lab/checks/fa-jsonl-export-lab/03-contract.sh.

Validate the output count cap

In /root/work/fa-jsonl-export-lab/service.py, validate_max(value) returns the value as it is only for an int from 1 to 1000 that is not a bool, and everything else is ValueError.

Require a cap from the caller so that an infinite input is not read to the end by mistake.

After saving, check with bash /opt/lab/checks/fa-jsonl-export-lab/04-contract.sh.

Consume only the rows you need

In /root/work/fa-jsonl-export-lab/service.py, take_rows(rows, maximum) is an iterator that lazily returns at most maximum rows, using islice or similar. It validates maximum when called, and each next consumes the input only once.

The moment you turn it into list(rows), you can no longer handle infinite input or very large input.

After saving, check with bash /opt/lab/checks/fa-jsonl-export-lab/05-contract.sh.

Serialize rows lazily

In /root/work/fa-jsonl-export-lab/service.py, json_lines(rows, maximum=100) yields encode_line for each row received from take_rows. It does not return a string or a list with everything combined.

Keep object selection and representation conversion each as a lazy stage.

After saving, check with bash /opt/lab/checks/fa-jsonl-export-lab/06-contract.sh.

Validate the downloaded lines again

In /root/work/fa-jsonl-export-lab/service.py, decode_lines(text) runs json.loads and then validate_row on each non-empty line of splitlines and returns a list. An empty string gives [], and an empty line in the middle is ValueError.

Distinguish an empty file from a malformed empty record.

After saving, check with bash /opt/lab/checks/fa-jsonl-export-lab/07-contract.sh.

Complete the HTTP download

In /root/work/fa-jsonl-export-lab/service.py, create_app(rows) returns json_lines(rows, 100) as an application/x-ndjson StreamingResponse at GET /export. rows is a list that can be iterated again. There must be no internal fields, and the content and order of each row must be preserved.

Check, together with a generator test, that the code does not just state streaming in the Content-Type while gathering everything internally.

After saving, check with bash /opt/lab/checks/fa-jsonl-export-lab/08-contract.sh.