TT Lab
Get started
Learn Learning paths Courses

Where Distributed Tracing Breaks

We joined producer and consumer as parent and child, and the trace never ended

Continue in TT Lab

Goal

With a small queue made of a single file, you first see in the dump the problem that arises when producing and consuming are connected as parent–child, and then change it to links. You attach batch consumption, queue wait time, span kinds, and link attributes in turn, apply it all the way to fan-out, and then follow the links to stitch one order's journey together beyond the trace.

Why it matters

Parent–child means "the parent waits for the child." But a producer does not wait for consumption to finish. If you connect the two as parent–child, the root span cannot close even though the response has already gone out to the user, and on a day when the queue backs up, one trace stays open for minutes. When fan-out gets mixed in, one order grows to thousands of spans and the backend starts giving that trace special treatment. Links exist for exactly this place — a relationship that keeps the causality but does not wait. If you set the consuming side up as the root of a new trace and point at the producing span with a link, each trace stays small, and the whole journey can be stitched back together by following the links. It is a different judgment from reading headers to join a broken chain or from passing context inside a process.

Steps

  1. Create /root/tp-links/naive.py. Read the dump path first from TRACELAB_OUT, and if it is absent use /root/tp-links/01-naive.jsonl. Put the queue file at 01-q.jsonl in the same directory as the dump and empty it at the start. Inside one order.submit span, put in 3 messages (m-1 to m-3) with bus.publish and wait in the queue with time.sleep(0.25), then inside the same span take them out with bus.poll, create a child span order.handle for each message, and call bus.handle. Then write four lines in /root/tp-links/01-problem.txt — after traces=, the number of traces in the dump, after root_span=, the root span's name, after wait_ms=, the value you get by subtracting the interval covered by the children from the root span's length, as an integer, and after problem=, what the problem with this shape is, in at least 40 characters.
  2. Create /root/tp-links/linked.py (default dump path /root/tp-links/02-linked.jsonl, queue file 02-q.jsonl). Inside order.submit, create a span order.publish for each message, and send that span's trace_id and span_id along in the message as hexadecimal strings. Run the side that takes messages out after waiting outside order.submit so that the span order.process becomes the root of a trace, and build a Link from the id that came in the message and attach it with links=. Both spans write the message id in the messaging.message.id attribute.
  3. Create /root/tp-links/batch.py (default dump path /root/tp-links/03-batch.jsonl, queue file 03-q.jsonl). This time put in 5 messages (m-1 to m-5), wait, and then take them out all at once with bus.poll(Q, 5). Process the whole batch you took out as a single span order.process.batch, attach as many links to that span as the number of messages taken out, and write the count in the integer attribute messaging.batch.message_count. This span must also be the root of a trace.
  4. Create /root/tp-links/wait.py (default dump path /root/tp-links/04-wait.jsonl, queue file 04-q.jsonl). Go back to the shape of step 2, but add a produced_at_ns field to the message and send the bus.now_ns() value along, and in the consuming span order.process, calculate the difference between that time and now in milliseconds and write it as the attribute messaging.queue.wait_ms. Then write one line per message in /root/tp-links/04-wait.tsv as <메시지 아이디><탭><기다린 밀리초 소수 첫째 자리> (the placeholders are the message id, a tab, and the milliseconds waited to one decimal place) (three lines).
  5. Create /root/tp-links/kinds.py (default dump path /root/tp-links/05-kinds.jsonl, queue file 05-q.jsonl). Attach span kinds to the step 4 flow — order.publish is SpanKind.PRODUCER, order.process is SpanKind.CONSUMER, and the wrapping order.submit is left as it is. And attach messaging.system (labbus), messaging.destination.name (orders), messaging.operation.type (send for producing, process for consuming), messaging.operation.name, and messaging.message.id to the two messaging spans. Then write three lines in /root/tp-links/05-kinds.tsv — each line is <스팬 이름><탭><종류><탭><operation.type> (the placeholders are the span name, a tab, the kind, a tab, and the operation.type) and the order is order.submit, order.publish, order.process, and the third column of order.submit is -.
  6. Create /root/tp-links/linkattrs.py (default dump path /root/tp-links/06-linkattrs.jsonl, queue file 06-q.jsonl). The same flow as step 5, but when you build a Link, pass attributes along as the second argument — write queue.message in link.relation, that message's id in messaging.message.id, and orders in messaging.destination.name. Each links field in the dump must contain these three attributes.
  7. Create /root/tp-links/fanout.py (default dump path /root/tp-links/07-fanout.jsonl, queue file 07-q.jsonl). Use the step 6 rules as they are, but extend the flow — order.submit creates 3 messages, and inside the order.process span that handles order A-1002 on the consuming side, it creates two more (span names invoice.publish and email.publish, message ids inv-2 and eml-2). After waiting a moment, take those two out and process them as invoice.process and email.process respectively; these too are roots of new traces and point at the preceding producing span with a link. The dump must contain 6 traces.
  8. Create /root/tp-links/journey.py. When run as python3 journey.py <덤프> <메시지아이디> (the placeholders are the dump and the message id), it starts from the order.publish span that created that message, follows the spans connected by links hop by hop, and prints one line <홉><탭><trace_id><탭><스팬이름> (the placeholders are the hop, a tab, the trace_id, a tab, and the span name) each. The starting span is hop 0, the next hop is the spans that point with a link at a span of the previous hop or a descendant of that span, and within the same hop they are printed in ascending order of span name. It stops when there is nowhere more to follow. Save the output of running this program on /root/tp-links/07-fanout.jsonl with m-2 in /root/tp-links/08-journey.tsv.

Notes

What happens if you connect producing and consuming as parent–child

Create /root/tp-links/naive.py. Read the dump path first from TRACELAB_OUT, and if it is absent use /root/tp-links/01-naive.jsonl. Put the queue file at 01-q.jsonl in the same directory as the dump and empty it at the start. Inside one order.submit span, put in 3 messages (m-1 to m-3) with bus.publish and wait in the queue with time.sleep(0.25), then inside the same span take them out with bus.poll, create a child span order.handle for each message, and call bus.handle. Then write four lines in /root/tp-links/01-problem.txt — after traces=, the number of traces in the dump, after root_span=, the root span's name, after wait_ms=, the value you get by subtracting the interval covered by the children from the root span's length, as an integer, and after problem=, what the problem with this shape is, in at least 40 characters.

The material is /opt/app/tracelab/tp_links/bus.py — it has publish, poll, handle, and now_ns. You get the uncovered interval with covered_ns(부모, 자식들) (the placeholders are the parent and the children) from /opt/lab/checks/_tplib.py. Run the instrumented program with /opt/otel-lab/bin/python, and delete the dump before making it again.

Set consuming up as the root of a new trace and connect with a link

Create /root/tp-links/linked.py (default dump path /root/tp-links/02-linked.jsonl, queue file 02-q.jsonl). Inside order.submit, create a span order.publish for each message, and send that span's trace_id and span_id along in the message as hexadecimal strings. Run the side that takes messages out after waiting outside order.submit so that the span order.process becomes the root of a trace, and build a Link from the id that came in the message and attach it with links=. Both spans write the message id in the messaging.message.id attribute.

Build the context with SpanContext(trace_id=..., span_id=..., is_remote=True, trace_flags=TraceFlags(TraceFlags.SAMPLED)) and wrap it with Link(ctx). You make the hexadecimal strings with format(값, "032x") and format(값, "016x") (the placeholder is the value), and to convert back you use int(문자열, 16) (the placeholder is the string). If the consuming loop remains inside the with order.submit block, it does not become a root, so look at the indentation.

Batch consumption as a single span with several links

Create /root/tp-links/batch.py (default dump path /root/tp-links/03-batch.jsonl, queue file 03-q.jsonl). This time put in 5 messages (m-1 to m-5), wait, and then take them out all at once with bus.poll(Q, 5). Process the whole batch you took out as a single span order.process.batch, attach as many links to that span as the number of messages taken out, and write the count in the integer attribute messaging.batch.message_count. This span must also be the root of a trace.

You give links= a list — build a Link for each message and pass them as a list. If you create a span per message, the one piece of work that processed the batch gets scattered, and if you create just one span with no links, you cannot tell which messages were in that batch. The shape that avoids both is this step's answer.

How do you measure the time waited in the queue

Create /root/tp-links/wait.py (default dump path /root/tp-links/04-wait.jsonl, queue file 04-q.jsonl). Go back to the shape of step 2, but add a produced_at_ns field to the message and send the bus.now_ns() value along, and in the consuming span order.process, calculate the difference between that time and now in milliseconds and write it as the attribute messaging.queue.wait_ms. Then write one line per message in /root/tp-links/04-wait.tsv as <메시지 아이디><탭><기다린 밀리초 소수 첫째 자리> (the placeholders are the message id, a tab, and the milliseconds waited to one decimal place) (three lines).

Processing is done one at a time in turn, so the later a message is taken out, the longer it waits — it is normal for the three values not to be equal. The time must be stamped by the producer, not the queue. If you take the moment the consumer took it out as the start, the time waited is always 0.

When to use PRODUCER and CONSUMER

Create /root/tp-links/kinds.py (default dump path /root/tp-links/05-kinds.jsonl, queue file 05-q.jsonl). Attach span kinds to the step 4 flow — order.publish is SpanKind.PRODUCER, order.process is SpanKind.CONSUMER, and the wrapping order.submit is left as it is. And attach messaging.system (labbus), messaging.destination.name (orders), messaging.operation.type (send for producing, process for consuming), messaging.operation.name, and messaging.message.id to the two messaging spans. Then write three lines in /root/tp-links/05-kinds.tsv — each line is <스팬 이름><탭><종류><탭><operation.type> (the placeholders are the span name, a tab, the kind, a tab, and the operation.type) and the order is order.submit, order.publish, order.process, and the third column of order.submit is -.

The table in the messaging semantic conventions decides the span kind by operation type — creating and sending are PRODUCER, and the place where the application processes a message is CONSUMER. The span that wraps the business flow is not a messaging span, so you do not change its kind. It is printed as is, by name, in the kind field of the dump.

Write on the link why it is connected

Create /root/tp-links/linkattrs.py (default dump path /root/tp-links/06-linkattrs.jsonl, queue file 06-q.jsonl). The same flow as step 5, but when you build a Link, pass attributes along as the second argument — write queue.message in link.relation, that message's id in messaging.message.id, and orders in messaging.destination.name. Each links field in the dump must contain these three attributes.

The second argument is the attributes, as in Link(ctx, {"키": "값"}) (the placeholders are the key and the value). With only the link, the fact that "they are connected" remains, but why they are connected does not — the same two spans could be connected because of a queue or because of a reprocessing, and to the person reading those are entirely different stories.

Apply it all the way to fan-out, where one creates many

Create /root/tp-links/fanout.py (default dump path /root/tp-links/07-fanout.jsonl, queue file 07-q.jsonl). Use the step 6 rules as they are, but extend the flow — order.submit creates 3 messages, and inside the order.process span that handles order A-1002 on the consuming side, it creates two more (span names invoice.publish and email.publish, message ids inv-2 and eml-2). After waiting a moment, take those two out and process them as invoice.process and email.process respectively; these too are roots of new traces and point at the preceding producing span with a link. The dump must contain 6 traces.

The second hop's producing span is a child of the consuming span — it is a synchronous call within the same trace, so parent–child is right. Jumping over with a link happens only when passing through a queue. Which relationship to connect with what is the whole of this step, and if you count the traces, you can tell right away whether they were split correctly.

Follow the links and stitch the journey back together

Create /root/tp-links/journey.py. When run as python3 journey.py <덤프> <메시지아이디> (the placeholders are the dump and the message id), it starts from the order.publish span that created that message, follows the spans connected by links hop by hop, and prints one line <홉><탭><trace_id><탭><스팬이름> (the placeholders are the hop, a tab, the trace_id, a tab, and the span name) each. The starting span is hop 0, the next hop is the spans that point with a link at a span of the previous hop or a descendant of that span, and within the same hop they are printed in ascending order of span name. It stops when there is nowhere more to follow. Save the output of running this program on /root/tp-links/07-fanout.jsonl with m-2 in /root/tp-links/08-journey.tsv.

To move to the second hop you have to look at descendants too — because the span that created the invoice message is a child of order.process, not order.process itself. The grader runs your program with other message ids too, so you must not hard-code m-2 in the code. Reading the dump does not need otel.