Reprocessing is unavoidable — what you stop is sending the order twice

In one line

When a process dies and comes back, you choose where to start reading again, and if you read from too early, orders go out twice, and if you read from too late, orders are missed. There is no safe point between the two. So the answer is not "read exactly once" but to make the same order id come out even if you read again, and have the receiver filter the second one.

Why this was needed

A program that sends orders usually has this shape. It reads one item from the signal stream, builds and sends an order message, and writes how far it has processed into a checkpoint. The whole problem is that the process can die between the three actions.

If you write the checkpoint after sending, then when it dies after sending but before writing, that stretch is read again. The order goes out again — that is at-least-once. If you write the checkpoint before sending, then when it dies after writing but before sending, that stretch is never read. The order is missed — that is at-most-once. On top of that, a checkpoint is usually not saved per item but in batches of several. So the overlapping stretch and the missing stretch are not one item wide but as wide as the batch size.

For orders the weights of the two are not equal. A missed order can just be placed again, but an order that went out twice may already have executed, and then an unwanted position arises. So the sending path usually chooses at-least-once and filters duplicates on the receiving side. Where to put deduplication is the real design decision of this problem.

How it works

To filter, you need a mark by which to recognize "the same order." An order message has an identifier attached by the sender, and in FIX-family specifications it is called ClOrdID. An amendment and a cancel attach a new id of their own and point to the previous id with OrigClOrdID. The meaning of the specification can be checked at FIXimate, and the specification itself on the FIX Trading Community standards page.

Here is the place where people trip most often. How do you make the id? A serial number counted from 1 on each run, a prefix mixing in the process id or the start time, and a random UUID — all three are common and all three are helpless against reprocessing. That is because reading the same signal again gives a different id. The receiver has no way to know whether it is a new order or the order it received a moment ago.

The fix is simple. Make the id from stable fields of the input only. If you join values that stay the same when read again, such as the stream name, offset, symbol, side, quantity, and price, and hash them, the same signal gives the same id however many times it is read again. You put in neither the time, nor random numbers, nor the process id.

And one more thing follows. The OrigClOrdID that amendments and cancels point to also must be recomputable by the same rule. If the table linking offsets to ids lives only in memory, it disappears on restart, and then a reprocessed cancel message points to an order that does not exist. The exchange rejects it, and meanwhile the original order is alive. The order that was meant to be canceled staying uncanceled is the real result when this chain breaks.

Finally, how long to make the deduplication window remains. This is not a matter of taste but an observed value. Measure from the data the difference between the time the same signal first went out and the time it went out again through reprocessing, and multiply by a margin. If the window is shorter than the maximum observed reprocessing delay, deduplication might as well not exist.

What it looks like in the field

Once I investigated double orders after an incident on a sending path. In the send log alone, the same offset had gone out twice, so everything looked like a duplicate, but when checked against the exchange's acknowledgments, half were not. One had been rejected for hitting the quantity limit, and one had been rejected because the cancel message pointed to an unknown order. What was actually dangerous was only the ones accepted on both sides, and their number was less than half of what was first counted. Counting by what was accepted, not by what was sent, is the starting point of the investigation.

Another thing. I often see incident reports that say only "we will add deduplication." Without a window length, that sentence is an opinion, not a task. If there is a delay measured from the data, it gets a number, and once it has a number, you can later ask whether that value was right.

The third thing you often meet is recovery done by hand. When an incident happens, the person in charge looks at the offset by eye and types in a value, and if that value is off by one digit, it becomes a double order as it is. If you cannot eliminate the hand-picked procedure, at least show on screen what that value means — if you read from this offset, how many items will go out again, and how many of those are orders already accepted. When those two numbers are visible, people usually pick the right value. Half of real incidents are recovery runbooks that wrote only the offset and not its impact.

And reprocessing does not end with orders. If the same signal flows twice, risk limit calculation, position aggregation, and audit logs also move twice together. If you deduplicate only the orders and leave the rest as is, you get a worse state where the order went out once but is booked twice in the books. The core of the design is to gather the deduplication point in one place and make every calculation that follows pass through that point.

What you will do in the next lab

You start with a signal stream of 48 items, the send log left by two runs, the exchange acknowledgments, the checkpoint, and the incident record. You find the reprocessing window, count the orders that went out again, implement a deterministic order id yourself, and then check against the acknowledgments to keep only the orders that were really accepted twice. Then you move the checkpoint save point before and after to count and compare duplicates and omissions, find the broken cancel and amendment chains, and finally derive the deduplication window length from the data and write a recovery runbook.