Reconnect a Broken Trace

Goal

A trace becomes useless as a whole if it is disconnected in even one place. But because nothing errors out when it breaks, nobody knows until they open the dashboard.

In this lab you are given three services that are already broken, and you connect them.

Getting started

cp /opt/lab/trace/* . && chmod +x run.sh
./run.sh            # a → b → c 로 요청 하나를 흘린다
python3 tree.py     # 이어졌는지 본다

Files

File	What it does
`svc.py`	One service. It has one bug
`run.sh`	Starts all three and sends one request through
`tree.py`	Reads the logs and judges whether they are connected

Good to know

run.sh does not use pkill -f svc.py to kill the services. That pattern also matches the command line of the shell that ran the script, so it kills itself. It writes down the PIDs and kills only those.

traceparent

00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│  │                                │                │
│  trace id (16바이트)               부모 span         플래그
버전

Steps

Confirm the break → 01-broken.txt
Where and why → 02-where.md
Connect it → the grader runs it itself to check
Read the format → 04-format.md
Sampling consistency → 05-sampling.txt
Concurrent requests → 06-concurrent.txt
Wrap-up → 07-notes.md

See the break with your own eyes

Copy /opt/lab/trace/ to a working directory, run ./run.sh, and save the result of python3 tree.py in 01-broken.txt.

Start with cp /opt/lab/trace/* . && chmod +x run.sh. One request flows through three services (a→b→c).

The symptom is the same as in the real world — each service sends its traces fine, but they do not connect. The dashboard fills up with single-span traces.

Pinpoint where and why it breaks

Read svc.py and write in 02-where.md which line is the problem. Also write why that one line breaks the trace.

Look at the place where it sends the request to the next service. If the receiving side has no header, it creates a new trace id — that is the correct behavior. The problem is the sending side.

The place where a trace breaks is almost always the calling side. No amount of staring at the receiving side's code will show it.

Connect them into one

Fix svc.py so that the three services share a single trace id and the parent–child relationships connect.

Attach the traceparent header to the outgoing request. The format is 버전-trace-내span-플래그 (the placeholders are the version, the trace, my own span, and the flags). What goes in the parent slot is not the parent you received but your own span right now — this is where people often get it wrong.

The grader runs it itself and checks for 1 kind of trace id, 1 root, and 0 spans whose parent cannot be found.

Read traceparent piece by piece

In 04-format.md, write what each of the four pieces of traceparent is, and give one actual value from the step 3 log as an example.

00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. From the front, they are the version, the trace id (16 bytes), the parent span (8 bytes), and the flags.

The lengths are fixed, so you can even verify it by eye — if the trace id is not 32 characters, it is wrong on the spot.

Sampling is decided only once

Even if the root service starts with the flag 00 (do not collect), check that all three services leave the same flag, and save it in 05-sampling.txt.

The decision is made once at the root and carried along as it is. If it is decided again in the middle, the trace becomes half a trace — nothing gets in the way of debugging as much as a trace whose front half was collected and whose back half is missing.

Change the starting flag at the root and run it, then save the flag line from tree.py.

Do they stay unmixed when several run at once

Send three requests at the same time and check that each forms its own complete trace separately, then save the result in 06-concurrent.txt.

After starting the services with run.sh as a reference, send three curl commands at the same time with &. If it is correct, there are 3 kinds of trace id, 3 spans in each trace, 3 roots, and 0 orphans.

The typical cause of breaking here is keeping the context in a global variable. It works fine when you test with one request, and the moment traffic arrives the traces get mixed together. This is why each language has a separate per-request storage.

Summarize the three points

Write at least three lines in 07-notes.md: where a trace breaks, what goes into the parent slot, and where the sampling decision is made.

The text must include 호출, 부모, and 샘플링 (the Korean words for call, parent, and sampling).