The cancel notice arrived twice and the position flipped sign

Goal

You attach the cancel and amendment notices for executions that have already gone out to the original executions, and instead of erasing the original ledger, you build a new ledger by adding offsetting records. You recompute the positions and compare them with the reports that already went out, confirm that the ledger is the same even when the same notice is received twice, and then produce the re-report targets and a correction report.

Why it matters

The execution ledger is data that has already flowed out to many places. If you quietly rewrite one line of it, yesterday's reports and today's ledger go out of line, and the reason for the mismatch is left nowhere. That is why financial books write afterward. A cancel is adding an offsetting record with the opposite sign, and an amendment is offsetting and then writing again with the corrected values. What is hard is not this rule itself but the edges. A notice may point to an execution that does not exist, the same notice may come twice through retransmission, and an amendment may come for an already-canceled execution. Missing even one makes the position flip to the opposite side, and that fact usually comes out only after the next day's report is wrong.

Steps

Use python3 to create fills.jsonl, notices.jsonl, position_report.csv, and rules.json in /root/bust/data.
Read the notices in the order received and write the decisions to /root/bust/notice_match.csv.
Add the cancels to be applied as offsetting records to create /root/bust/ledger_bust.jsonl.
Add the amendments as offsets and rebookings to create /root/bust/ledger_final.jsonl, and output the average price per order to /root/bust/order_avg.csv.
Recompute the positions and write only the mismatched symbols to /root/bust/position_diff.csv.
Feed the same notice stream through twice to create /root/bust/ledger_twice.jsonl and /root/bust/idem.json.
Gather the notices that arrived after the report time to create /root/bust/rereport.csv.
Write the source of each and every change to /root/bust/trace.csv and the overall summary to /root/bust/report.json.

Notes

For notices, seq is the arrival order. Make the decisions in this order, and you have to remember the cancels you applied to be able to tell a re-cancel.
There are four decision names. A notice whose identifier was already seen earlier is 중복통지 (the Korean word for "duplicate notice"); if the execution it points to is not in the ledger, 대상없음 (the Korean word for "no target"); if the execution it points to was already canceled earlier, 이미취소됨 (the Korean word for "already canceled"); and the rest are 적용 (the Korean word for "applied"). Look in this order.
A ledger entry has eleven fields: entry_id, kind, fill_id, order_id, symbol, side, qty, px, notional, source_notice, entry_time. kind is one of the three fill, reversal, rebook.
The entry_id of an offsetting record is the notice identifier with -rv attached, and that of a rebooking has -rb attached. entry_time is the notice time.
If new_qty or new_px of an amendment notice is null, use the original execution's value for that one as it is.
Compute amounts with decimal and round at the fourth decimal place with ROUND_HALF_UP. Write them as strings.
Count positions by adding buys and subtracting sells. Offsetting records have negative quantities, so the sign flips by itself.
The position report time is in the data file rules.json. Do not use date to write today.
Common mistake 1: amending by fixing the original record. The pre-application data must remain so that you can compare before and after.
Common mistake 2: not remembering the notice identifier and so offsetting a retransmitted cancel twice.
Public documents: FIX Standards, Fiximate, decimal. The eighteen executions and fourteen notices are synthetic data for this lab.

Generate the execution ledger, the cancel and amendment notices, and the position report

Use python3 to create fills.jsonl, notices.jsonl, position_report.csv, and rules.json in /root/bust/data. Use the generation script as is, which uses no random numbers.

We cannot bring in the customer's execution ledger as it is, so we build synthetic data of the same shape. Without random numbers, the same data comes out no matter who runs it how many times, and you can compare each other's judgments. The grader converts the data to a canonical form and compares fingerprints, so if you edit it by hand, all the later steps get blocked.

Attach notices to the original executions and judge them

Put the first line seq,notice_id,type,target_fill_id,verdict in /root/bust/notice_match.csv, read the notices in the order received, and write one of 적용, 대상없음, 이미취소됨, or 중복통지 (the four Korean words mean applied, no target, already canceled, and duplicate notice). Write every notice on one line.

Look at four things in this order. Was the same notice identifier already seen earlier, is the execution it points to in the ledger, was that execution already canceled earlier, and the rest is applied. Every time you apply a cancel, you must remember that execution identifier for the third check to be possible. Remember every notice identifier regardless of the decision.

Add the cancels as offsetting records to build a new ledger

In /root/bust/ledger_bust.jsonl, copy the eighteen original executions as they are as records whose kind is fill, and for each cancel to be applied, add a reversal record whose quantity and amount have the opposite sign. Do not modify the original records.

Add without deleting. The offsetting record's quantity is the original execution quantity with a minus sign, and the unit price is the original execution's as it is. That way the amount also gets the opposite sign by itself. If you attach -rv to the notice identifier for the entry_id and leave that notice in source_notice, you can follow the source later. The source_notice of an original execution record is an empty string.

Reflect the amendments and recompute average prices

/root/bust/ledger_final.jsonl is the step 3 ledger with two lines, reversal and rebook, added for each amendment. Put the first line order_id,net_qty,net_notional,avg_px in /root/bust/order_avg.csv and write one line per order.

An amendment offsets the original record and then writes one line again with the corrected values. Use the original execution's value as it is for values not in the notice. An amendment that changes only the price has the same quantity and only a different amount. The average price is the ledger's sum of amounts divided by the sum of quantities, and for a fully canceled order the sum of quantities is 0, so set it to 0. Check for 0 before dividing.

Recompute positions and compare with the existing report

Put the first line symbol,reported_qty,recomputed_qty,diff in /root/bust/position_diff.csv and write only the symbols whose position recomputed from the final ledger differs from position_report.csv.

Count positions by adding buys and subtracting sells. Offsetting records have negative quantities, so the sign flips by itself and you do not need a separate branch. diff is the recomputed value minus the reported value. Do not write symbols whose values are the same. Not all the differences are explained by late-arriving notices, so if there are symbols that are not explained, leave them as they are.

Test whether the ledger is the same even when you receive the same notice twice

Attach the same notices once more after the notice stream (add 1000 to seq in the second batch) to create /root/bust/ledger_twice.jsonl, and write entries, same_as_final, and ignored_duplicates to /root/bust/idem.json.

Retransmission is normal. If you do not remember the notice identifier, an execution that was canceled once gets offset twice and the position flips to the opposite side. Every identifier in the second batch has been seen earlier, so none of them should be applied, and then the ledger becomes exactly the same as the step 4 one. ignored_duplicates is the number of notices judged to be duplicate notices in the whole attached stream, and same_as_final is true or false.

Gather notices that arrived after the report as re-report targets

Put the first line notice_id,type,target_fill_id,symbol,position_delta in /root/bust/rereport.csv and write those among the applied notices whose notice time is later than the position report time in rules.json.

A notice that arrives after the position report has gone out is not finished by fixing only the books; it makes the report that already went out wrong. The reference time is in rules.json, so do not use today's date. position_delta is how much that notice moves the position, and for a sell the sign is opposite. An amendment that changes only the price does not touch the quantity, so it becomes 0, but do not drop it from the re-report targets.

Write the correction report and the change trace table

Put the first line entry_id,kind,fill_id,source_notice,reason,qty,px,notional in /root/bust/trace.csv and write every record in the final ledger whose kind is not fill, and write the overall summary in /root/bust/report.json.

A report is useful only if the person who receives it can decide what to do next. If you attach to each change which notice it came from and that notice's reason, you can follow it months later. Take reason from the notice data, but it must be that of the notice that was actually applied. A copy retransmitted with the same identifier has a different reason, and that is where they diverge. The keys to put in the summary are fills, notices, 적용, 대상없음, 이미취소됨, 중복통지, bust_applied, adjust_applied, entries_before, entries_after, position_mismatch, and rereport (the four Korean keys are the same four decision names as in step 2).