Rerun a Past Range Without Counting It Twice

Goal

You build a runner runner.py that takes an interval and reruns the past. It does partition-level replacement with the same code as the regular run, prevents collisions with the regular run through interval reservation, seals intervals that have already gone out and leaves corrections, and directly reproduces and then fixes the cumulative aggregate doubling because of a backfill.

Why it matters

A backfill is the job of rerunning a past interval, but the hard part is not the computation but the coordination. While the backfill runs, the regular run also runs. If the two write the same date at the same time, nobody knows which one won. The idempotency lab in the same course was about making the result the same even if the same row is inserted twice. Here it is the interval, not the row. Even if the partition table is idempotent, if the cumulative table is not idempotent, the number swells with one backfill. An additive cumulative total is certainly wrong when it meets a backfill, and a derived cumulative total gives the same value no matter how many times you run it. And there are places you must not go back to. A report already sent and an alert already issued are not data but events. You seal that interval, and when a new value comes out, you must record it separately as a correction instead of overwriting, so that you can say both "what did we send at that time" and "what is correct now." The grader does not trust your text. It sets up the partitions that the grader created in a temporary directory, points to that store with environment variables, actually runs your runner, and opens the sqlite tables directly to compare. The dates and amounts change on every run.

Steps

1. Create and run /root/backfill/gen_events.py to create fourteen days of partitions from dt=2026-02-01.jsonl to dt=2026-02-14.jsonl under /root/backfill/events.
2. In /root/backfill/runner.py, create init and run <시작일> <끝일> <주인> so that it counts the partitions of the interval and puts them into daily.
3. Fix run so that it replaces partitions wholesale, and write the answer from running everything at once, the answer from running day by day, and the rerun result in /root/backfill/split.json.
4. Add rollup-add <시작일> <끝일> and rollup, and write in /root/backfill/double.json how far the two methods diverge after a backfill.
5. Add claim <시작일> <끝일> <주인> and release <시작일> <끝일> <주인> so that it reserves intervals.
6. Make run skip partitions that someone else holds and report them as skipped. If anything was skipped, the exit code is 5.
7. Add seal <시작일> <끝일> and amend <날짜> <사유>. run does not touch sealed partitions, and a voucher that arrived late is left in corrections as a correction.
8. Leave a single page with /root/backfill/backfill_report.json and /root/backfill/backfill_report.md.

Reference

• Execution contract: python3 /root/backfill/runner.py <명령> ... (the placeholder stands for the command). Give the answer as a single JSON object on standard output. The exit codes are 0 success, 2 an unknown command or a wrong number of arguments, 3 no state store, 4 a reservation was denied, 5 there was a skipped interval, and 6 amend was called on an interval that is not sealed.
• The state store path must be changeable with the environment variable BACKFILL_DB and the source partition folder with BACKFILL_EVENTS. The defaults are /root/backfill/state.db and /root/backfill/events, respectively. The grader points to its own store with these two variables.
• A partition file name is dt=YYYY-MM-DD.jsonl, and the line shape is {"order_id": 문자열, "dt": 날짜, "amount": 정수, "status": "paid" 또는 "refund"} (a string, a date, an integer, and "paid" or "refund"). The amount of a refund line is negative. A partition's orders is the line count and amount is the sum of amounts.
• There are five tables. daily(dt, orders, amount, owner) · rollup(metric, value) · claims(dt, owner) · seals(dt) · corrections(dt, orders_before, amount_before, orders_after, amount_after, reason). Do not put a primary key on daily — implementing the replacement yourself is the point of this lab.
• run response: {"owner": 문자열, "done": [날짜...], "skipped": [[날짜, 사유]...], "orders": 정수, "amount": 정수} (a string, dates, pairs of date and reason, and integers). orders and amount are the sums of only the partitions in done. The reasons are the three claimed_by:<주인> (the placeholder stands for the owner) · sealed · no_data. A date with no source file (no_data) does not make the exit code 5.
• claim response: {"owner": 문자열, "claimed": [날짜...], "denied": [[날짜, 주인]...]} (a string, dates, and pairs of date and owner). A date you already hold goes into claimed as is. release response: {"owner": 문자열, "released": [날짜...]} (a string and dates), and it releases only your own reservations.
• seal response: {"sealed": [날짜...]} (dates). amend response: {"dt": 날짜, "orders_before": 정수, "amount_before": 정수, "orders_after": 정수, "amount_after": 정수, "reason": 문자열} (a date, integers, and a string). amend does not fix daily and adds one line to corrections.
• The rollup-add response and the rollup response are {"mode": "add" 또는 "derive", "order_total": 정수, "amount_total": 정수} ("add" or "derive", and integers). rollup-add adds the interval sum to the existing value, and rollup recomputes from the whole of daily and overwrites.
• Do not judge by performance. Do not measure how long the backfill took; leave only what you touched in what order.
• Official documentation: python sqlite3 · SQLite Transaction · SQLite UPSERT · Airflow Dag Runs
• Common mistakes: building separate code for the backfill, putting the whole interval in one transaction, silently overwriting someone else's interval, leaving the cumulative total as the additive method, and overwriting an interval that has already gone out and losing what was sent at that time.
• To see the state store with your eyes, use sqlite3 /root/backfill/state.db 'SELECT * FROM daily ORDER BY dt'.

Create fourteen days of partitions

Create and run /root/backfill/gen_events.py to create files from dt=2026-02-01.jsonl to dt=2026-02-14.jsonl under /root/backfill/events. One day is one file, and the amount of a refund line is negative.

You can advance the dates with datetime.date.fromisoformat and timedelta. If you vary the number of lines per day, the per-partition sums can later be told apart. Mix in a few refund lines so that the amount is not a simple running total.

Build a runner that takes an interval

In /root/backfill/runner.py, create init and run <시작일> <끝일> <주인> (the placeholders stand for the start date, the end date, and the owner). run counts the lines and the amount for each partition of the interval and puts them into daily, and a date with no source file is left in skipped as no_data.

A regular run is just the case where the interval is one day. So build only one function. Create the tables in init with CREATE TABLE IF NOT EXISTS, and do not put a primary key on daily. no_data does not make the exit code 5.

Replace at the partition level

Fix run so that when it reruns a partition, it deletes that partition's old result and then inserts. Then write the answer from running everything at once, the answer from running day by day, and the rerun result in /root/backfill/split.json as whole, by_day, and rerun_changed, and refill daily in /root/backfill/state.db so that there is one line per date.

Replacement is deleting that date's rows and inserting again. To compare the answer from running everything at once with the answer from running day by day, you have to create the store fresh twice, and if you point to a temporary store with the BACKFILL_DB environment variable, you can compare without touching the main store.

Reproduce the cumulative aggregate doubling

Add rollup-add <시작일> <끝일> (the placeholders stand for the start and end dates) and rollup. Then, in a temporary store, run the whole interval and add it to the cumulative total, backfill part of the interval and add that interval again, and write how far it diverges from the derived value in /root/backfill/double.json as backfill_range, true_total, add_after_backfill, and gap.

rollup-add adds the interval sum to the existing value, and rollup recounts the whole of daily and overwrites. The size of the gap must equal exactly the amount sum of the backfill interval — because that interval went into the cumulative total twice.

Reserve the interval first

Add claim <시작일> <끝일> <주인> and release <시작일> <끝일> <주인> (the placeholders stand for the start date, the end date, and the owner). Put a date that someone else holds into denied along with the owner, and the exit code is 4. release releases only your own reservations.

If you make the date the primary key of the claims table, two owners cannot hold one date. Treat holding again a date you already hold as a success, not a denial — if a retry looks like a failure, nobody retries.

Skip other people's intervals and report

Make run leave a partition someone else holds untouched and record it in skipped as claimed_by:<주인> (the placeholder stands for the owner). If anything was skipped, the exit code is 5. Partitions you hold yourself are run as they are.

The point is to neither wait nor overwrite. If you skip and say so, the caller can decide whether to call again or to call a person. You only need to read the reservation list once before walking through the interval.

Leave intervals that already went out as corrections

Add seal <시작일> <끝일> and amend <날짜> <사유> (the placeholders stand for the start date, the end date, the date, and the reason). run skips a sealed partition as sealed. Then add one late-arriving voucher line to the 2026-02-02 partition, seal that date, and leave a correction with amend. The before and after of the correction must differ.

amend does not fix daily. It only writes the value recounted from the source and the value remaining in daily side by side in corrections. If amend is called on a date that is not sealed, reject it with exit code 6 — such a date you can just run.

Report one rerun on a single page

In /root/backfill/backfill_report.json, write partitions, orders, amount, sealed, corrections, rollup, and double_gap, and in /root/backfill/backfill_report.md, write four sections: ## 무엇을 다시 돌렸나 ## 정기 실행과 어떻게 부딪혔나 ## 누적 집계는 왜 두 배가 되나 ## 되돌리면 안 되는 자리 (in order: what was rerun, how it collided with the regular run, why the cumulative aggregate doubles, and places you must not go back to).

partitions, orders, and amount are read from the daily table, sealed from seals, corrections as a line count, and rollup from the derived cumulative total. Use the value measured in step 4 as is for double_gap. In the report, write the amount of the gap as a number.