Handling Gaps and Latency in a Market Data Feed

Goal

You tell what is missing from what arrived late in a market data feed capture, separate what A/B arbitration can fill from the real holes, measure latency with percentiles and pinpoint the windows where the tail is concentrated, and decide the recovery method for each window.

Why it matters

A market data feed runs on a transport that guarantees neither ordering nor arrival. So the receiver has to judge by sequence number, and if the judging method is wrong, the mistake does not pass quietly; it clogs the line. If you count gaps by arrival order, you get more than five times the real number, and if you request retransmissions for that many, the request traffic grows latency and latency grows reordering, so even more gaps appear.

The four things this lab enforces are all rules from the field. Judge gaps as a set — a late message is not a missing one, and you cannot tell the two apart by arrival order. Look at the other channel before retransmitting — that is why the redundancy exists, and the share that gets filled without a request is overwhelming. Look at latency by the tail — the mean mixes two worlds into one number. Write the calculation method in a file — if the percentile definitions differ, the same data leads to different conclusions, and from then on it is a discussion about definitions, not data.

This capture is about 11 minutes right after the open on 2026-08-27. The original the exchange sent has no gaps from sequence number 1 to 12000, and we received it on two channels, A and B. Each channel suffered losses, and latency spiked sharply right after the open and around 09:08.

Percentiles are computed like this — nearest rank. Sort the values in ascending order and take the idx = ceil(p * n / 100)-th value (counting from 1). In Python integers that is lat[(p*n + 99)//100 - 1]. The mean is the sum divided by the count, rounded down. The latency analysis covers channel A only, and if the same seq arrived several times, use only the one that arrived first.

Steps

Create /root/cap/feed, then run the answer key's generator as is to produce /root/cap/feed/feed.jsonl. If you change the seed, the values will not match the grading values.
Write seven lines to /root/cap/feed/shape.txt: lines=, ch_a_lines=, ch_b_lines=, ch_a_unique=, ch_b_unique=, ch_a_dup=, seq_max=.
Write arrival_jumps_a=, missing_a=, missing_b=, gap_runs_a= to /root/cap/feed/gaps.txt, and write the sequence numbers channel A did not receive to /root/cap/feed/gaps_a.csv, grouped into contiguous ranges, with the header from_seq,to_seq,count.
Write recoverable_from_b=, true_holes=, hole_runs= to /root/cap/feed/arb.txt, and write the ranges neither side received to /root/cap/feed/holes.csv as from_seq,to_seq,count.
Write method=nearest_rank and n=, p50_us=, p95_us=, p99_us=, max_us=, mean_us= to /root/cap/feed/latency.txt.
Split into buckets by the minute of ts_send, write minute,n,mean_us,p99_us,over_10ms to /root/cap/feed/tail.csv, and write worst_minute=, worst_p99_us=, over_10ms_total=, burst_minutes= to /root/cap/feed/tail.txt. minute is an integer with 09:00 as 0, and burst_minutes joins, with commas, the minutes whose mean exceeds 10 ms.
For each true-hole range, write from_seq,to_seq,count,action to /root/cap/feed/retx.csv. If it is 5 or more in a row, the action is snapshot; if fewer, retransmit. Write retransmit_runs=, retransmit_seqs=, snapshot_runs=, snapshot_seqs= to /root/cap/feed/retx.txt.
Write a report in /root/cap/feed/report.md. It needs five sections: ## 무슨 일이 있었나, ## 갭, ## 지연, ## 돈으로 얼마인가, ## 무엇을 고쳐야 하나 (the five Korean section titles mean: what happened, gaps, latency, what it amounts to in money, and what to fix).

Notes

The timestamps go down to the microsecond. If you read them by slicing the string, you never go through floating point, so it is safe.
The minute buckets in this capture are eleven, from 0 to 10.
Common mistake 1: counting gaps in step 3 by scanning the file in arrival order. That number is not a target for retransmission requests.
Common mistake 2: not removing duplicates in step 5. If you count the ones that arrived twice through retransmission, the distribution wobbles.
Common mistake 3: mixing in channel B in step 6. The two channels take different paths, so their latency distributions differ too.

Generate the A/B redundant feed capture

Create /root/cap/feed, then run the answer key's generator as is to produce /root/cap/feed/feed.jsonl. If you change the seed, the values will not match the grading values.

Just run the generator as it is. If you change the seed, the values will not match the grading values.

Count the lines per channel and the duplicates

Write seven lines to /root/cap/feed/shape.txt: lines=, ch_a_lines=, ch_b_lines=, ch_a_unique=, ch_b_unique=, ch_a_dup=, seq_max=.

Count the number of lines per channel and the number of distinct seq values separately. If the two numbers differ, the difference is the duplicates.

Tell what is missing from what is late

Write arrival_jumps_a=, missing_a=, missing_b=, gap_runs_a= to /root/cap/feed/gaps.txt, and write the sequence numbers channel A did not receive to /root/cap/feed/gaps_a.csv, grouped into contiguous ranges, with the header from_seq,to_seq,count.

Count in two ways. One is reading in arrival order and counting the places where seq jumps by more than 1, and the other is counting, out of 1..12000, the sequence numbers that channel never received at all.

Fill from channel B and leave only the true holes

Write recoverable_from_b=, true_holes=, hole_runs= to /root/cap/feed/arb.txt, and write the ranges neither side received to /root/cap/feed/holes.csv as from_seq,to_seq,count.

Subtract what B received from the set of sequence numbers A did not receive, and what remains is the true holes. Then group those back into contiguous ranges.

Compute latency percentiles

Write method=nearest_rank and n=, p50_us=, p95_us=, p99_us=, max_us=, mean_us= to /root/cap/feed/latency.txt.

Use channel A only, and only the one that arrived first for each seq. The percentile is nearest rank — sort ascending and take the ceil(p*n/100)-th value. Round the mean down with integer division.

Pinpoint the windows where the tail is concentrated

Split into buckets by the minute of ts_send, write minute,n,mean_us,p99_us,over_10ms to /root/cap/feed/tail.csv, and write worst_minute=, worst_p99_us=, over_10ms_total=, burst_minutes= to /root/cap/feed/tail.txt. minute is an integer with 09:00 as 0, and burst_minutes joins, with commas, the minutes whose mean exceeds 10 ms.

Split into buckets by the minute of ts_send, and for each bucket produce the count, the mean, the p99, and the count over 10 ms. A minute whose mean exceeds 10 ms is a burst.

Split into retransmission and snapshot

For each true-hole range, write from_seq,to_seq,count,action to /root/cap/feed/retx.csv. If it is 5 or more in a row, the action is snapshot; if fewer, retransmit. Write retransmit_runs=, retransmit_seqs=, snapshot_runs=, snapshot_seqs= to /root/cap/feed/retx.txt.

Sort the true-hole ranges from step 4 by length. If it is 5 or more in a row, it is snapshot; if fewer, retransmit.

Write the feed quality report

Write a report in /root/cap/feed/report.md. It needs five sections: ## 무슨 일이 있었나, ## 갭, ## 지연, ## 돈으로 얼마인가, ## 무엇을 고쳐야 하나 (the five Korean section titles mean: what happened, gaps, latency, what it amounts to in money, and what to fix).

It needs five sections. The latency section must include p99, and the remediation section must say which of the gap judgment method, the monitoring metric, or duplicate removal to change.