Three Clocks Mean Three Orderings

In one line

The exchange time, the gateway receive time, and the application processing time are different clocks, and depending on which clock you sort by, the answer to "did the cancel come first or the fill?" flips.

Why this was needed

An event usually has two or more timestamps stamped on it. At first you do not see why there are so many, and you just pick one and sort by it. That choice changes the conclusion.

ts_exch   거래소가 그 일이 일어났다고 찍은 시각
ts_recv   우리 게이트웨이가 그 메시지를 받은 시각
ts_proc   우리 애플리케이션이 그것을 처리한 시각

The gaps between the three timestamps vary with the network path and the queue length. The problem is that there is more than one path. If the fill notice and the cancel response pass through different sessions, different lines, and different processes, something that happened earlier at the exchange reaches us later.

How it works

Sort the order from the previous lesson by two different clocks, and the answers split like this.

거래소 시각 순서            수신 시각 순서
  cancel_req                  cancel_req
  fill        (누적 300)      cancelled
  cancelled                   fill        (누적 300)
--------------------------  --------------------------
마지막 이벤트 = cancelled   마지막 이벤트 = fill
결론: 취소된 주문           결론: 체결된 주문

Same six lines, same folding rule, different answers. And this difference is not rare. It is common for the cancel response to be a short message that arrives quickly while the fill notice takes a relatively slower path, so when a cancel and a fill race within a few milliseconds, the order flips almost every time.

Which one should you rely on? The order in which things actually happened in the market is the exchange time. Our receive time is a fact about our infrastructure, not a fact about the market. That is why post-hoc reconstruction, audits, and dispute handling sort by exchange time. If two events have the same timestamp, break the tie with the sequence number the exchange provided. With only one sort key, the same input gives a different answer each time.

Still, you do not throw the receive time away. The difference between the two timestamps is the latency, and it is used to find the stretches where latency spikes. Also, a real-time processor cannot see the future, so it has no choice but to process events in the order they arrive. The realistic answer is to accept that the real-time result and the post-hoc reconstruction can differ, and to set up a procedure in which the post-hoc reconstruction is the correct answer and the real-time result is corrected against it.

The best design is one where the result does not depend on the order. In this case, changing the rule as follows makes the answers from the two clocks the same: once replay is finished, if the cumulative fills are at least the order quantity, the state is filled. The justification lies in the domain. A fill is an irreversible fact, and a cancel applies only to the remaining quantity. A cancel arriving for an order whose remaining quantity is 0 is normal, and it does not erase the fill.

What it looks like in the field

When a dispute arises with the exchange, both sides bring their own logs. The exchange submits a table sorted by its own time, and we submit one sorted by ours. If the two tables disagree on order, from that point on it is no longer a matter of establishing facts but of which clock to recognize, and contracts and regulations usually write the exchange time as the reference. A log design that keeps only our time and discards the exchange time therefore leaves us defenseless in a dispute.

Another thing: a single server with bad NTP can hold up an investigation for weeks. At one site, the clock on one of two gateways ran about 400 ms ahead, and only the orders that passed through that machine were recorded as having the cancel before the fill. In the data it looked concentrated in particular symbols and particular time windows, so people spent a long time looking for market factors. In an investigation that deals with time, suspecting the clock itself first saves time.

What you will do in the next lab

Sort the same event log by exchange time and by receive time to make two final-state tables, and pinpoint exactly which orders flip their conclusion. Then add a rule that does not depend on order and confirm that the two tables become the same.