Idempotently Reviving a Dead End-of-Day Batch

Goal

You revive an end-of-day close that died midway by re-running it, finishing with neither double posting nor omission. And you gain the hands to choose the close targets by telling the accounting date apart from the system time.

Why it matters

A bank system has the time the clock shows (posted_at) and the date the books recognize (biz_date), separately. The accounting date advances only when the close batch finishes, so for transactions that come in while the close is running, it becomes ambiguous which date to post them to. Incidents that arise here are a large share of bank batch failures.

A close batch must be idempotent. It means the result must be the same no matter how many times you run it for the same accounting date. Otherwise, rerunning when the batch dies becomes a new failure in itself. There are only two ways to re-run, and the incremental way of adding only what has not been done on top of what is already reflected presumes that "everything already reflected is correct." If that presumption is broken, the incremental way has no means to pull out what went in wrongly.

This case goes like this. The 2026-08-25 end-of-day close batch started at 22:10 and died midway. Of the 240 journal lines for accounting date 2026-08-25, 153 are reflected, and the balance sum of the closing output is not 0. On top of that, while the close was running, one online channel kept stamping the old accounting date. The cutoff is 2026-08-25T22:10:00 and the next business day is 2026-08-26.

Steps

Create and run /root/bank/gen_eod.py to make /root/bank/eod.db. It holds 372 journal lines and the close history.
Write seven lines in /root/bank/eod_state.txt. business_date= and eod_status= come from sys_param, txn_total=, applied=, and unapplied= from txn for that accounting date, and balance_rows= and closing_sum= from daily_balance.
Dump the lines whose posted_at is after the cutoff and whose biz_date is 2026-08-25 into /root/bank/misdated.csv with the header txn_id,posted_at,biz_date,eod_applied. And write three lines, misdated=, misdated_applied=, and naive_after_cutoff=, in /root/bank/misdated.txt. The last value is the number of lines when you filter by posted_at alone without looking at the accounting date.
Move the biz_date of those lines to 2026-08-26 and set eod_applied back to 0. Do not touch 2026-08-24, whose close is already finished.
Write a plan in /root/bank/rerun_plan.md with four sections: ## 두 가지 재수행 방식, ## 선택과 근거, ## 멱등성을 어떻게 보장하나, and ## 중단 기준 (the Korean headings mean: The two re-run approaches, Choice and rationale, How idempotency is guaranteed, and Stop criteria). The Choice and rationale section must include the number of lines to recompute as a number.
Create /root/bank/eod_rerun.sh and run it once, and dump the output into /root/bank/db_run1.csv with the header biz_date,account,debit_total,credit_total,closing_balance. Also leave a success record in eod_run.
Run the same script once more and dump /root/bank/db_run2.csv, then write five lines in /root/bank/idempotency.txt: run1_rows=, run1_closing_sum=, run2_rows=, run2_closing_sum=, and diff=.
Roll eod_status in sys_param over to CLOSED and business_date to 2026-08-26, and write a report in /root/bank/eod_report.md with five sections: ## 무엇이 멈췄나, ## 계정일자가 왜 어긋났나, ## 재수행을 어떻게 안전하게 만들었나, ## 검증, and ## 남은 위험 (the Korean headings mean: What stopped, Why the accounting date went off, How the re-run was made safe, Verification, and Remaining risks).

Notes

If you do not hard-code the cutoff value and read it with SELECT value FROM sys_param WHERE name='eod_cutoff', there is less room for the conditions to drift apart.
Whether the sum of the closing output's closing_balance is 0 is the fastest health signal. If it is not 0, there are lines where only one side of an entry is reflected.
Common mistake 1: filtering by posted_at alone in step 3. Transactions of other channels that correctly stamped the next day come along with it.
Common mistake 2: changing only biz_date in step 4 and leaving eod_applied as it is. Those lines stay in the previous close's output and are skipped in the next close.
Common mistake 3: writing the step 6 script with INSERT alone. The values double on the second run.

Reproduce the moment the close died

Create and run /root/bank/gen_eod.py to make /root/bank/eod.db. It holds 372 journal lines and the close history.

Build /root/bank/eod.db with python3. There are four tables, sys_param, txn, daily_balance, and eod_run, and the journal lines are 372.

Count how far the dead close got

Write seven lines in /root/bank/eod_state.txt. business_date= and eod_status= come from sys_param, txn_total=, applied=, and unapplied= from txn for that accounting date, and balance_rows= and closing_sum= from daily_balance.

Choose the close targets by biz_date, not posted_at. Also look at whether the sum of the output's closing_balance is 0 — if it is not 0, only one side of an entry is reflected.

Find the transactions stamped with the wrong accounting date

Dump the lines whose posted_at is after the cutoff and whose biz_date is 2026-08-25 into /root/bank/misdated.csv with the header txn_id,posted_at,biz_date,eod_applied. And write three lines, misdated=, misdated_applied=, and naive_after_cutoff=, in /root/bank/misdated.txt. The last value is the number of lines when you filter by posted_at alone without looking at the accounting date.

There are two conditions. The lines whose posted_at is after the cutoff and, at the same time, whose biz_date is the closing day. If you filter by the first condition alone, transactions of other channels that correctly stamped the next day come along too.

Move the accounting date to the next business day

Move the biz_date of those lines to 2026-08-26 and set eod_applied back to 0. Do not touch 2026-08-24, whose close is already finished.

You must not change only biz_date. Lines already reflected in the closing output are mixed in, so eod_applied must be reverted too. Do not touch the previous day, whose close is already finished.

Decide the re-run approach and write the rationale

Write a plan in /root/bank/rerun_plan.md with four sections: ## 두 가지 재수행 방식, ## 선택과 근거, ## 멱등성을 어떻게 보장하나, and ## 중단 기준 (the Korean headings mean: The two re-run approaches, Choice and rationale, How idempotency is guaranteed, and Stop criteria). The Choice and rationale section must include the number of lines to recompute as a number.

You need four sections. In the choice and rationale section, write as a number how many lines are to be recomputed, and in the idempotency section, write which table you handle and how.

Build the re-run script and run it once

Create /root/bank/eod_rerun.sh and run it once, and dump the output into /root/bank/db_run1.csv with the header biz_date,account,debit_total,credit_total,closing_balance. Also leave a success record in eod_run.

Do not type SQL by hand; make it a script. The second run must be literally the same as the first for you to test idempotency. And dump the first run's output as a CSV.

Run it once more to prove idempotency

Run the same script once more and dump /root/bank/db_run2.csv, then write five lines in /root/bank/idempotency.txt: run1_rows=, run1_closing_sum=, run2_rows=, run2_closing_sum=, and diff=.

Run the same script again as it is and dump the output again. The result of comparing the two CSVs line by line is diff. If the values doubled, you added without deleting.

Confirm the close and roll over the accounting date

Roll eod_status in sys_param over to CLOSED and business_date to 2026-08-26, and write a report in /root/bank/eod_report.md with five sections: ## 무엇이 멈췄나, ## 계정일자가 왜 어긋났나, ## 재수행을 어떻게 안전하게 만들었나, ## 검증, and ## 남은 위험 (the Korean headings mean: What stopped, Why the accounting date went off, How the re-run was made safe, Verification, and Remaining risks).

Producing the output and confirming the close are different things. Roll over the close state and the accounting date in sys_param together. And write the five sections of the report.