TT Lab
Get started
Learn Learning paths Courses

Where Distributed Tracing Breaks

The span is there, but nothing reproduces

Continue in TT Lab

Goal

Starting from one line of a failed request's span, you fill in attributes and events until you could reproduce the same failure holding only that span. You go the full circle: how to convert values that must not be included, how to count a distinct-value budget, and how to write a team convention in a file and enforce it with a checker.

Why it matters

Auto-instrumentation fills in the HTTP surface. The path, the status code, and the length come out, but if you open that span at dawn and ask "so what do we put in to make this happen again," there is no answer. You have to leave the reproduction input, the state at that time, and what we did as attributes before an investigation can begin. Conversely, if you put in the whole request body, contact details and tokens pile up as they are in the observability backend — so instead of the original you keep hashes, categories, and lengths. A fact that has a point in time is an event, not an attribute, and a relationship that is not parent–child is a link. Finally, if you do not count how many distinct values each key has, a single exception message inflates a key into thousands of values and kills the search screen.

Steps

  1. Read one line of /opt/app/tracelab/tp_attrs/failed.jsonl. It is the span of a failed request, but it lacks the values needed for reproduction. Write five lines in /root/tp-attrs/01-missing.txt — each line is <열쇠이름>=<왜 필요한가> (the placeholders are the key name and why it is needed), and the keys are, in order, shop.cart.item_count, shop.request.body_bytes, shop.cache.hit, shop.queue.depth, and shop.payment.retry_count. For each key, write the reason in at least 25 characters, in your own words, answering "what can we not judge without this value?"
  2. Create /root/tp-attrs/02_attrs.py. While processing order ord-1010 with tracelab.tp_attrs.orders, create one SERVER span named POST /checkout and attach six attributes: the five keys from step 1 plus the order number shop.order.id. For payment, try charge(주문번호, 시도번호) (the placeholders are the order number and the attempt number) up to three times, as 1, 2, and 3, and shop.payment.retry_count is (the number of attempts − 1). The default dump path is /root/tp-attrs/02-attrs.jsonl, and if the environment variable TRACELAB_OUT is set, use that instead.
  3. Create /root/tp-attrs/03_redact.py. Add three attributes to step 2 — shop.customer.email_hash is the first 16 characters of the SHA-256 hexadecimal string of the email, shop.payment.card_brand is the card's brand value, and shop.auth.token_len is the length (an integer) of the auth token. The originals of the email, card number (pan), token, and postal code are not left in any attribute. The default dump path is /root/tp-attrs/03-redact.jsonl.
  4. Create /root/tp-attrs/04_events.py. On top of step 3, leave (1) one cache.miss event if the cache missed, and (2) one payment.attempt.failed event each time a payment attempt fails — the event attributes are attempt (the integer attempt number) and reason (the kind of PaymentError). (3) If all three attempts fail, put the exception message as it is in the shop.error.message attribute and change the span status to ERROR. The default dump path is /root/tp-attrs/04-events.jsonl. You will look at this step's shop.error.message again in step 5.
  5. Create /root/tp-attrs/05_bulk.py and process the 200 entries of orders.ORDER_IDS once each with the same instrumentation (one span per request). The default dump path is /root/tp-attrs/05-bulk.jsonl. Then write one line for each attribute key that appears in the dump in /root/tp-attrs/05-cardinality.tsv as <열쇠><탭><고유값수><탭><판정> (the placeholders are the key, a tab, the number of distinct values, a tab, and the verdict), in ascending order of key name. The verdict is free for shop.order.id and shop.customer.email_hash, low for anything else with 20 or fewer distinct values, and leak if it exceeds 20.
  6. Create /root/tp-attrs/convention.tsv. Each line is one key, with four tab-separated columns, <열쇠><탭><타입><탭><허용값><탭><고유값성격> (the placeholders are the key, a tab, the type, a tab, the allowed values, a tab, and the nature of the distinct values). The type is one of string, int, bool, or deny, the allowed values are * (no restriction) or a list joined with |, and the nature of the distinct values is free or low. Of the ten keys that appeared in step 5, block shop.error.message with deny and instead add a new shop.error.kind with declined|timeout. Also add in advance shop.refund.amount (int, free), which you will use in step 8, and write the original-forbidden keys shop.customer.email, shop.payment.card_pan, and shop.auth.token as deny as well. For deny lines, leave the allowed values and nature columns as -.
  7. Create /root/tp-attrs/lint_spans.py. When called as python3 lint_spans.py <규약파일> <덤프> (the placeholders are the convention file and the dump), it prints VIOLATION <열쇠> <이유> (the placeholders are the key and the reason) once per key, in ascending order of key name, for each key that breaks the convention and exits with code 1, and if nothing is broken it prints one line starting with OK and exits with code 0. There are four things to check — a key that is not in the convention, a key blocked with deny, a value of a different type, and a value not in the allowed list. In addition, if a key declared low has more than 20 distinct values in the dump, that is a violation too. After building it, run it on /root/tp-attrs/05-bulk.jsonl and on /opt/app/tracelab/tp_attrs/noisy.jsonl respectively, and write two lines in /root/tp-attrs/07-violations.tsv as <덤프파일이름><탭><깨진 열쇠들을 쉼표로 이은 것> (the placeholders are the dump file name, a tab, and the broken keys joined with commas), in that order.
  8. Create /root/tp-attrs/08_refund.py and make one POST /refund SERVER span that handles refund ord-1027 triggered by the queue. The attributes are shop.order.id, shop.refund.amount, shop.queue.depth, shop.customer.email_hash, shop.payment.card_brand, shop.auth.token_len, and shop.payment.retry_count, and if it ends in failure, add shop.error.kind and set the status to ERROR (do not put in the original message). For each failed attempt, leave a payment.attempt.failed event, and attach the trace coordinates that orders.job_context(주문번호) (the placeholder is the order number) gives as a link (do not attach it as the parent). The default dump path is /root/tp-attrs/08-refund.jsonl, and when you run the step 7 checker on this dump, OK must come out.

Notes

Take a failed span and write down the missing values

Read one line of /opt/app/tracelab/tp_attrs/failed.jsonl. It is the span of a failed request, but it lacks the values needed for reproduction. Write five lines in /root/tp-attrs/01-missing.txt — each line is <열쇠이름>=<왜 필요한가> (the placeholders are the key name and why it is needed), and the keys are, in order, shop.cart.item_count, shop.request.body_bytes, shop.cache.hit, shop.queue.depth, and shop.payment.retry_count. For each key, write the reason in at least 25 characters, in your own words, answering "what can we not judge without this value?"

To lay out the dump nicely, python3 -m json.tool /opt/app/tracelab/tp_attrs/failed.jsonl is handy. All the span has now is the HTTP surface — path, status code, and length. Ask yourself whether you could reproduce the same failure from that alone. The grader also checks that the five keys are really absent from that span.

Add the values needed for reproduction as attributes

Create /root/tp-attrs/02_attrs.py. While processing order ord-1010 with tracelab.tp_attrs.orders, create one SERVER span named POST /checkout and attach six attributes: the five keys from step 1 plus the order number shop.order.id. For payment, try charge(주문번호, 시도번호) (the placeholders are the order number and the attempt number) up to three times, as 1, 2, and 3, and shop.payment.retry_count is (the number of attempts − 1). The default dump path is /root/tp-attrs/02-attrs.jsonl, and if the environment variable TRACELAB_OUT is set, use that instead.

orders.payload returns the request body, orders.body_bytes its size, and orders.cache_lookup and orders.queue_depth the state at that time. A payment failure is orders.PaymentError. Run the instrumented program with /opt/otel-lab/bin/python — the system python3 does not have OpenTelemetry.

Convert values that must not be included into hashes, categories, and lengths

Create /root/tp-attrs/03_redact.py. Add three attributes to step 2 — shop.customer.email_hash is the first 16 characters of the SHA-256 hexadecimal string of the email, shop.payment.card_brand is the card's brand value, and shop.auth.token_len is the length (an integer) of the auth token. The originals of the email, card number (pan), token, and postal code are not left in any attribute. The default dump path is /root/tp-attrs/03-redact.jsonl.

The reason you keep a converted form instead of throwing the original away entirely is that there are still questions you can answer — with a hash, "does it repeat for the same user"; with a category, "does it happen only with a particular card brand"; with a length, "did the token arrive truncated." You make the hash with hashlib.sha256(문자열.encode("utf-8")).hexdigest() (the placeholder is the string).

Move facts that have a point in time into events

Create /root/tp-attrs/04_events.py. On top of step 3, leave (1) one cache.miss event if the cache missed, and (2) one payment.attempt.failed event each time a payment attempt fails — the event attributes are attempt (the integer attempt number) and reason (the kind of PaymentError). (3) If all three attempts fail, put the exception message as it is in the shop.error.message attribute and change the span status to ERROR. The default dump path is /root/tp-attrs/04-events.jsonl. You will look at this step's shop.error.message again in step 5.

That there were two retries is a single number, so it is an attribute, and when and why the first retry happened is a record with a timestamp, so it is an event. The two do not compete — things you count are attributes, the moment something happened is an event. You change the status with span.set_status(Status(StatusCode.ERROR, "...")).

Count the distinct values per key and build a budget table

Create /root/tp-attrs/05_bulk.py and process the 200 entries of orders.ORDER_IDS once each with the same instrumentation (one span per request). The default dump path is /root/tp-attrs/05-bulk.jsonl. Then write one line for each attribute key that appears in the dump in /root/tp-attrs/05-cardinality.tsv as <열쇠><탭><고유값수><탭><판정> (the placeholders are the key, a tab, the number of distinct values, a tab, and the verdict), in ascending order of key name. The verdict is free for shop.order.id and shop.customer.email_hash, low for anything else with 20 or fewer distinct values, and leak if it exceeds 20.

An identifier is normal when it differs on every request, and a categorical key is normal when it has only a few values. If an original leaks into a slot that should be a category, one key comes to have thousands of values — in this dump one leak comes out, and it is the very attribute you deliberately put in at step 4.

Write the team convention in a file

Create /root/tp-attrs/convention.tsv. Each line is one key, with four tab-separated columns, <열쇠><탭><타입><탭><허용값><탭><고유값성격> (the placeholders are the key, a tab, the type, a tab, the allowed values, a tab, and the nature of the distinct values). The type is one of string, int, bool, or deny, the allowed values are * (no restriction) or a list joined with |, and the nature of the distinct values is free or low. Of the ten keys that appeared in step 5, block shop.error.message with deny and instead add a new shop.error.kind with declined|timeout. Also add in advance shop.refund.amount (int, free), which you will use in step 8, and write the original-forbidden keys shop.customer.email, shop.payment.card_pan, and shop.auth.token as deny as well. For deny lines, leave the allowed values and nature columns as -.

If you leave a convention only as sentences, the next person will not read it. You have to make it a table a machine can read before you can attach a checker. The reason to write the forbidden keys as well is that if the agreement "let's not put those in" lives only in code review, it leaks in the end. It has fifteen lines.

Build a checker that catches spans that break the convention

Create /root/tp-attrs/lint_spans.py. When called as python3 lint_spans.py <규약파일> <덤프> (the placeholders are the convention file and the dump), it prints VIOLATION <열쇠> <이유> (the placeholders are the key and the reason) once per key, in ascending order of key name, for each key that breaks the convention and exits with code 1, and if nothing is broken it prints one line starting with OK and exits with code 0. There are four things to check — a key that is not in the convention, a key blocked with deny, a value of a different type, and a value not in the allowed list. In addition, if a key declared low has more than 20 distinct values in the dump, that is a violation too. After building it, run it on /root/tp-attrs/05-bulk.jsonl and on /opt/app/tracelab/tp_attrs/noisy.jsonl respectively, and write two lines in /root/tp-attrs/07-violations.tsv as <덤프파일이름><탭><깨진 열쇠들을 쉼표로 이은 것> (the placeholders are the dump file name, a tab, and the broken keys joined with commas), in that order.

If you print violations per span, 200 lines come out — collect them and print once per key. noisy.jsonl is another team's dump, so you cannot fix it. What you do at this step is "make a machine tell you what is off," and the fixing is done in step 8 with your own code.

Instrument a second handler by the convention and connect it with a link

Create /root/tp-attrs/08_refund.py and make one POST /refund SERVER span that handles refund ord-1027 triggered by the queue. The attributes are shop.order.id, shop.refund.amount, shop.queue.depth, shop.customer.email_hash, shop.payment.card_brand, shop.auth.token_len, and shop.payment.retry_count, and if it ends in failure, add shop.error.kind and set the status to ERROR (do not put in the original message). For each failed attempt, leave a payment.attempt.failed event, and attach the trace coordinates that orders.job_context(주문번호) (the placeholder is the order number) gives as a link (do not attach it as the parent). The default dump path is /root/tp-attrs/08-refund.jsonl, and when you run the step 7 checker on this dump, OK must come out.

You build the link with Link(SpanContext(trace_id=int(16진문자열, 16), span_id=int(16진문자열, 16), is_remote=True, trace_flags=TraceFlags(0x01))) (the placeholders are hexadecimal strings) and pass it with start_as_current_span(..., links=[link]). If you attach the queue job as the parent, you get a strange parent that started hours ago and ends now — the place where there is a relationship but it is not parent–child is exactly a link. If the checker prints VIOLATION, fix the instrumentation, not the convention.