TT Lab
Get started
Learn Learning paths Courses

The Language of Banking

Turning an Incoming File Away Before You Load It

Continue in TT Lab

Goal

You build a gatekeeper, filegate.py, that checks fixed-width and CSV inbound files sent by counterparty institutions before loading and rejects the whole file if anything is off. It leaves its judgment as a machine-readable report, an exit code, and a rejection file to return to the counterparty.

Why it matters

Finding wrong lines after loading is always more expensive than rejecting at the door. The lines that already went in have been picked up by other batches, and undoing them brings correcting entries and apology calls. Even if 998 items are fine, if 2 items deviate from the specification, not a single line of that file is loaded. If you load only half, the counterparty's trailer and our ledger never match. The width of a fixed-width file is bytes, not character count. One Korean character is 2 bytes in CP949 and 3 bytes in UTF-8, so when the encoding changes, the line length itself changes. In CSV, commas and line breaks inside quotes must not be counted as delimiters. The grader does not trust your wording. It lays out inbound files it made itself in a temporary directory, runs your script, and compares the judgments and error codes. The institution codes, counts, and amounts change on every run.

Steps

  1. Create and run /root/bankfile/gen_inbound.py to make 5 day's files from four counterparty institutions in /root/bankfile/inbound/.
  2. Make /root/bankfile/filegate.py check the skeleton of CSV inbound files and match the trailer count and total, and emit a report and an exit code.
  3. Add fixed-width (.txt) handling to filegate.py. Measure whether a line is exactly 80 bytes by bytes, not character count.
  4. Make filegate.py reject files that come in an encoding different from the specification with ENC. Write the encoding read for each file in the report.
  5. Fix filegate.py's CSV parsing to follow RFC 4180. Quoted commas and line breaks are not field delimiters.
  6. Add field-level validation to filegate.py, and leave a rejection file for each rejected file. If there is even one error, not a single line of that file is loaded.
  7. Make filegate.py reject a resend with the same institution and the same sequence number as DUP. If the hash is the same, it is resend; if different, it is conflict.
  8. Process your own inbox and leave /root/bankfile/gate_report.json, /root/bankfile/rejected/, and /root/bankfile/receipt.md.

Notes

Build a day's inbox

Create and run /root/bankfile/gen_inbound.py to make 5 files in /root/bankfile/inbound/. They are a fixed-width (CP949) file of 120 records and its retransmission that resends it byte for byte, a CSV of 90 records (with 3 or more lines whose recipient name contains a comma), a CSV of 60 records whose trailer count is 1 less than the actual, and a file that is fixed width but arrived in UTF-8.

The name is IN-<YYYYMMDD>-<기관코드 6자리>-<꼬리>.txt|.csv (the placeholders are the date, the 6-digit institution code, and the suffix). The width of fixed width is bytes, so you must measure with text.encode('cp949') and then pad with spaces. For the retransmission, just write the same bytes once more under a different name. Enclose CSV names containing commas in double quotes.

Start by matching the trailer's count and total

Make /root/bankfile/filegate.py check the H, D, and T skeleton of CSV inbound files and match the trailer's count and total against the values it actually counted. It emits a report and exit codes 0 and 2.

A count mismatch and a total mismatch are different incidents, so give them separate codes (TRAILER_COUNT and TRAILER_TOTAL). The records and total in the report are not the values written in the trailer but the values you counted yourself from the D records. The grader tests with a different institution code and count each time.

The width of fixed width is bytes, not characters

Add .txt fixed-width handling to /root/bankfile/filegate.py. If a line is not exactly 80 bytes in CP949, reject with WIDTH and leave the number of the line that is off.

len(line) is the character count, and what the specification talks about is len(line.encode('cp949')). The cutting positions for fields are also on bytes, not the string. Only lines where a Korean name was padded by character count deviate in width.

Block files that arrive in an encoding different from the specification

Make /root/bankfile/filegate.py read .txt as CP949 and .csv as UTF-8, and reject with ENC if it cannot be decoded in that encoding. Write the encoding read for each file in the report's encoding.

bytes.decode() raises a UnicodeDecodeError on failure. If you write the exception's start and reason in detail, the counterparty's contact knows at which byte it broke. The gatekeeper must not automatically switch encodings to read.

A comma inside quotes is not a delimiter

Fix /root/bankfile/filegate.py's CSV parsing to follow RFC 4180. The records and total must be right for files with quoted commas and line breaks, and a file whose quote is not closed must be rejected.

The standard library csv module implements the quoting rules as they are. To pass a string, wrap it in io.StringIO. If a quote is not closed, all the following lines get sucked into one field and the field count does not match.

Field validation, rejection files, and no partial loading

Add validation of transaction number, recipient name, account, and amount to /root/bankfile/filegate.py, and leave a <거절디렉터리>/<파일이름>.reject.csv (the placeholders are the rejection directory and the file name) with the header line,code,detail for each rejected file. If there is even one error, that file must not go into the accepted list.

The transaction number must be unique within the file, and if it overlaps, point to the later line. Fields cut out of fixed width are looked at with left and right spaces stripped. The rejection file must be readable as it is by the counterparty institution's batch, so emit it as a table, not as human sentences.

Tell apart a file that came again and a file that conflicted

Make /root/bankfile/filegate.py reject as DUP a file with the same institution and sequence number as one already accepted. If the file hash is the same, write resend in detail, and if different, write conflict, and leave the sha256 in the report.

The sequence number is in the header record. You must remember only files accepted earlier - a rejected file was not loaded, so a later one with the same number is not a duplicate. Compute the hash over the whole file bytes with hashlib.sha256.

Process your own inbox and produce a receipt report

Process your own inbox to leave /root/bankfile/gate_report.json and /root/bankfile/rejected/, and write five sections, ## 수신 요약, ## 거절한 파일과 사유, ## 재전송으로 판정한 파일, ## 적재하지 않은 이유, and ## 상대 기관에 요청할 것, in /root/bankfile/receipt.md (the Korean headings mean: Receipt summary, Rejected files and reasons, Files judged as retransmissions, Why they were not loaded, and What to request from the counterparty institution).

Use the /root/bankfile/filegate.py you built in the earlier steps as it is. In the summary, write the number accepted, the number rejected, and the sum of amounts that will actually be loaded, as numbers. Write the names of rejected files exactly so the counterparty can find them. The grader rereads /root/bankfile/inbound and compares against your report, so do not write the report by hand; use what filegate.py produced.