The book on the screen had been quietly wrong all along

Goal

You rebuild the order book from one snapshot and incremental updates, and count in price levels the difference made by the places where the sequence broke and by lines that arrived late. At the end you confirm that an implementation with the resync rule does not differ from the reference snapshot by even one level.

Why it matters

A quote screen makes no sound even when it is wrong. Values do not go empty or throw exceptions; a slightly different value keeps circulating, so only days later, after orders go out strangely, do you trace back. The two-line rule — how you join the sequence number stamped in the snapshot to the increments' sequence numbers, and whether you stop or keep applying when the sequence breaks — makes that difference. And the calculations in this lab all deal with prices. The moment you subtract prices in binary floating point, the test that measures one tick quietly becomes 0, so you also look at the cases where you lose first at the data type rather than at the value.

Steps

1. Use python3 to create four data files in /root/cap/book/data. Use the generation script as is.
2. Count the shape of the increment stream and write it as nine lines in /root/cap/book/shape.txt.
3. Build the order book from the start snapshot, count the overlapping increments, and write it in /root/cap/book/init.json.
4. Apply the increments in sequence-number order and write the book and the counts by action in /root/cap/book/naive.json.
5. Find the places where an invariant breaks during replay and write them as seven lines in /root/cap/book/invariant.txt.
6. Write the difference between the broken stretches and the rebuilt result in /root/cap/book/gap.json.
7. Write the lines that arrived late and the difference they made in /root/cap/book/late.json.
8. Leave the final order book with the resync rule in /root/cap/book/final/book.json and the quality report in /root/cap/book/final/report.json.

Notes

• The feed format is written in /root/cap/book/data/feed_spec.txt. It is not a real exchange specification but a synthetic format for this lab.
• The session constants are in /root/cap/book/data/session.json. The tick size, the sequence number at which the snapshot was taken, the start snapshot, and the reference snapshot are all there, so do not hard-code numbers in the code.
• Handle prices with decimal.Decimal. If you subtract with float, one tick does not come out exactly, and the spread test in step 5 falls to 0.
• Count the number of mismatched price levels by turning each of the two books into a set of (종목, 방향, 가격, 수량) (the placeholders are symbol, side, price, and quantity) and taking the size of the symmetric difference. If only the quantity at one price differs, one shows up on each side, so it is 2.
• The line order of updates.jsonl is the arrival order. It may differ from the sequence-number order.
• Common mistake 1: storing quantity 0 as a value. 0 is not a value but an action meaning delete that price level.
• Common mistake 2: picking the last snapshot as the one to rebuild from in step 6. Use the earliest checkpoint among those that can cover the break.
• Standards documents: FIX Trading Community, FIXimate, decimal module.

Generate the snapshot and the increment stream

Use python3 to create the four files snapshots.jsonl, updates.jsonl, session.json, and feed_spec.txt in /root/cap/book/data. Use the generation script as is, which uses no random numbers.

An air-gapped network has no market data capture to download, so you generate the data yourself first. Without random numbers, the same data comes out no matter who runs it how many times, and you can compare each other's reconstruction results. The grader converts the data to a canonical form and compares fingerprints, so if you edit the data by hand, all the later steps get blocked.

Count the shape of the increment stream

Write the nine lines updates=, snapshots=, symbols=, seq_min=, seq_max=, seq_missing=, delete_ops=, upsert_ops=, out_of_order= to /root/cap/book/shape.txt.

seq_missing is the number of integers from seq_min to seq_max that are not in the file. delete_ops is the lines with quantity 0, and upsert_ops is the lines that are not 0. Do not try to split add from change here — those two can be told apart only when you hold the order book. out_of_order is the number of lines whose sequence number is smaller than an earlier line's.

Build the order book from the snapshot and discard the overlapping increments

Build each symbol's order book from the start_checkpoint checkpoint snapshot in session.json, and in /root/cap/book/init.json write, per symbol, the nine values snap_id, last_seq, bid_levels, ask_levels, best_bid, best_ask, discarded, applicable, and first_applied_seq.

Because increments keep going out while the snapshot is being taken, last_seq is carried along with it. Increments at or below that value are already in the snapshot and must be discarded. discarded is the number of lines discarded that way, and applicable is the number of lines that remain; both count only that symbol's lines. best_bid is the highest bid and best_ask is the lowest ask.

Apply the increments and reconstruct the order book

Apply to the step 3 order book all the increments whose sequence number is greater than start_checkpoint, in ascending sequence-number order, and in /root/cap/book/naive.json write the two keys book and ops. In ops, put the five counts add, change, delete, delete_missing, and applied.

The action is decided not by the message but by the order book you are holding right now. Quantity 0 means delete, but the price you are trying to delete may not be in the book. Do not treat that as an error; count it as delete_missing — that number itself is a signal that there are increments you did not receive. The broken sequence numbers are simply passed over in this step.

Find the moment a crossed order book appears

While doing the same replay as in step 4, check that symbol's invariant after each update, and write the seven lines first_cross_seq=, first_cross_symbol=, first_cross_bid=, first_cross_ask=, crossed_states=, tick_spread_states=, zero_qty_levels= to /root/cap/book/invariant.txt.

A cross is a state in which the best bid is greater than or equal to the best ask. crossed_states is the number of times that symbol was crossed right after applying, and tick_spread_states is the number of times it was not crossed and the spread was exactly one tick. If the second value comes out as 0, suspect the data type rather than the calculation — subtracting decimal fractions in binary floating point does not give exactly one tick.

Compare what kept applying at the break with what was rebuilt

In /root/cap/book/gap.json write the four keys gaps (the list of broken stretches), resync (the list of places rebuilt), level_mismatch (the number of mismatched price levels per symbol), and level_mismatch_total. A gaps entry is from_seq, to_seq, count, and a resync entry is detected_after_seq, next_seq, checkpoint.

A broken sequence means there is a change you did not receive, and there is no way to know what it was. So do not apply more increments; rebuild from the next snapshot. The checkpoint to rebuild from is the earliest one at or above the sequence number applied up to just before the break, and increments at or below that snapshot's last_seq must be discarded again. Write both lists from the smaller sequence number first.

What happens if you apply lines that arrived late as they are

In /root/cap/book/late.json write the four keys late_seqs (the list of sequence numbers of lines placed late), harmful_seqs (those among them that change the result), level_mismatch (the number of mismatched price levels per symbol), and level_mismatch_total. The comparison is between the sequence-order replay of step 4 and a replay under the same conditions that applies the lines in the order they appear in the file.

Not every line placed late is harmful. The old value comes back to life only when a larger sequence number has overwritten the same symbol, side, and price first. For lines that are not like that, the result is the same even if the order changes. Keep both replays identical in passing over the broken places and in discarding lines at or below the start checkpoint — that way the difference comes only from the arrival order.

Compare against the reference with the final implementation that has the resync rule

Run the whole stream with an implementation that has both rules (apply in sequence-number order, and rebuild from the next snapshot when it breaks), and in /root/cap/book/final/book.json write the final order book and in /root/cap/book/final/report.json write the eleven values applied, discarded, resync_count, resync_checkpoints, gap_count, lost_seqs, late_seqs, naive_vs_reference, final_vs_reference, crossed_symbols_final, and reference_checkpoint.

The reference order book is the reference_checkpoint checkpoint snapshot in session.json. naive_vs_reference is the number of mismatched price levels between the step 4 book and the reference, and final_vs_reference is that between this book and the reference, and the latter must be 0. discarded is every line discarded because its sequence number had already passed — count the initially overlapping range and the range overlapping after the rebuild together. Do not delete the earlier steps' deliverables; leave them in a new directory.