Observing LLM Serving with OpenTelemetry: Actual Signals from vLLM, The Distance from Standards, and Filling the Gap with a Collector

Status: Draft (October 7, 2026; October 9, 2026, collector overhead measurement reinforced). All numerical figures in this article were measured directly by the author. The environment consists of a single 8GB-class GPU, vLLM 0.31.0, Qwen3-1.7B, OpenTelemetry Collector contrib 0.162.0, and a single server. Because the GenAI semantic conventions are still in the Development stage , their names are changing (even on the day of writing this article, one metric name changed). Tracing for SGLang and Dynamo, as well as trace storage (Tempo, Jaeger, etc.) and metric storage, are Unmeasured . Writing rule: Measured data is the default, and anything unmeasured is marked as "Unmeasured".

Table of Contents

• 1. Configuration
• 2. Signals Emitted by vLLM
  • 2.1 Traces: One llm_request per request
  • 2.2 Metrics: Prometheus /metrics instead of OTLP
• 3. Overhead of Enabling
• 4. Distance from Standards and Filling the Gap with a Collector
• 5. How to Find Slow Requests
  • 5.1 Separating Time to First Token Using Only Span Attributes
  • 5.2 Tail-Based Sampling vs. Probabilistic Sampling
• 6. Experiences and Issues
• 7. OTCA Scope and Further Reading
• 8. Limitations and Unmeasured Items
• References

When an LLM server slows down, you generally want to know three things: how long the request waited in the queue, how long the calculation took, and which service call initiated this request. OpenTelemetry (OTel) is the standard for recording these three things in the same way, and the collector, sampling, semantic conventions, and context propagation tested in the OTCA are all used here. This article attaches these to a single vLLM instance and answers five questions.

1. What spans and attributes does vLLM leave for a single request, and what names do the metrics appear under?
2. Does the trace initiated at the client continue all the way to vLLM?
3. What is the overhead of enabling it?
4. How different is it from the standard (GenAI semantic conventions), and can it be filled using a collector?
5. How does sampling that keeps only slow requests work, and how is it different from probabilistic sampling?

Short answer. For each request, a llm_request span remained attached to the traceparent , and simply using its attributes allowed us to separate time-to-first-token into waiting and computation. The overhead of enabling tracing was within the margin of error. Although the attribute names differed from the standard (such asprompt_tokens vs. input_tokens), they could be changed using the collector's transform processor. Tail-based sampling, which keeps only slow requests, kept all 19 slow requests, whereas 10% probabilistic sampling kept only 1 out of 22 slow requests.

1. Configuration

클라이언트(계측, traceparent 전송) ──HTTP──▶ vLLM(--otlp-traces-endpoint) ──OTLP/gRPC──▶ OTel Collector ─▶ 파일(JSON)
                                              └─ /metrics(Prometheus) ◀──스크레이프── Collector(prometheus 수신기)

• vLLM 0.31.0 vllm serve Qwen/Qwen3-1.7B --otlp-traces-endpoint http://127.0.0.1:4317. The OpenTelemetry SDK (1.40.0) and OTLP exporter were already included within the same image.
• For the collector, the contrib distribution binary version 0.162.0 was run within the same pod. The receivers were OTLP (gRPC) and prometheus, and the exporter was a file exporter (one JSON line per batch).
• Storage backends (Tempo, Jaeger, Prometheus, etc.) were not attached. This article only covers "what is emitted" while "where it is stored and how it is viewed" is Unmeasured .
• Scripts and raw data are located in the repository's docs/experiments/2026-10-07-otel-llm-serving/ .

2. Signals Emitted by vLLM

2.1 Traces: One llm_request per request

The client created a parent span and sent three requests carrying W3C traceparent headers. All three vLLM spans that arrived at the collector were llm_request(SERVER kind), their parent span IDs matched the client span's ID , and their trace IDs matched as well. In other words, the trace started at the client continued into vLLM.

Each span had twelve attributes, and changing the configuration still resulted in the same twelve attributes.

Absences are also important. The model name (gen_ai.request.model) and gen_ai.operation.name were absent from this span, and the resource's service.name was unknown_service . If a service name is not assigned, spans from multiple servers get mixed under a single name. You can assign it via the environment variable OTEL_SERVICE_NAME or populate resource using the collector's service.name processor (we used the latter in Section 4).

The server startup process is also recorded as traces. llm_request Aside from Overall Loading , dozens of startup spans emerged as a single trace. As averages across four servers where model files were already present, EngineCoreProc init was 18.7 seconds, Prepare model was 6.7 seconds, Executor init was 6.1 seconds, Load model was 3.4 seconds, Warmup (GPU) was 2.7 seconds, and Load weights was 1.8 seconds (spans contain parent-child relationships, so they should not be summed). For a freshly booted server, downloading weights took 298 seconds and Overall Loading was 341 seconds. You can read "why the server started slowly" through span durations instead of digging through logs.

2.2 Metrics: Prometheus /metrics

instead of OTLP /metricsIn this version, metrics were only confirmed via vllm: (Prometheus format). When fetched every 5 seconds using the collector's prometheus receiver, 93 names appeared, and the useful ones are below (all starting with the

We checked three things.

• _created series leak out as metrics. Prometheus format _created lines were converted into gauges by the receiver, making nearly four out of ten names (36) useless. Dropping metric_relabel_configs with the scrape configuration .*_created reduced it to 57.
• Histograms are cumulative and have empty units. The TTFT histogram created by the receiver had a aggregationTemporality of cumulative (2) and an unit that was empty (the standard is seconds s), and the labels were engine and model_name . The sum after a load of 120 requests was 51.76 seconds, making the average 0.43 seconds, but the bucket distribution (49 cases of 0.25–0.5s, 19 cases of 0.5–0.75s, 7 cases of 1–2.5s) shows that more fall above the average.
• Colons (vllm:...) in metric names are preserved as is.

3. Cost of Enabling

The same load (512 inputs, 128 outputs, concurrency 1, 8, 32) was measured across three configurations, restarting the server twice for each. Since the KV cache placement could vary each time the server started (Part 14), server logs were checked, and all six servers showed 26,560 tokens.

• Even with tracing enabled, the throughput and time to first token were identical within 1%. The cost of creating one span per request was negligible compared to a single request for this model (about 15ms per token). For smaller models or faster token speeds, the ratio might increase, but this was not measured (unmeasured).
• --collect-detailed-traces all did not increase spans or attributes in this version). Request spans for 500 requests collected from four servers (two with tracing on, two with verbose on) all shared the same twelve attributes. Although the option description states that it "may impact performance," no difference was observed in this test.
• Storage Volumewas 612KB for 500 spans in JSON (about 1.2KB per span). This is the JSON size of file exports and does not represent OTLP transmission size or storage size.

4. Distance from the Standard, and Bridging with a Collector

OpenTelemetry's GenAI semantic conventions have been moved to a separate repository (semantic-conventions-genai) and their status is development . Comparing the contents at the time of verification (latest commit as of 2026-10-06) with what vLLM outputs yields the following.

Since the standard is still under development, rather than either side being "wrong," vLLM is simply using names from before the standard was finalized. For users, having a collector handle the translation avoids the need to modify the server. This was verified by actually running the configuration below.

processors:
  resource/service:
    attributes:
      - {key: service.name, value: vllm, action: upsert}
  transform/genai:
    trace_statements:
      - context: span
        statements:
          - set(attributes["gen_ai.usage.input_tokens"], attributes["gen_ai.usage.prompt_tokens"]) where attributes["gen_ai.usage.prompt_tokens"] != nil
          - set(attributes["gen_ai.usage.output_tokens"], attributes["gen_ai.usage.completion_tokens"]) where attributes["gen_ai.usage.completion_tokens"] != nil
          - delete_key(attributes, "gen_ai.usage.prompt_tokens")
          - delete_key(attributes, "gen_ai.usage.completion_tokens")
  metrics_transform/genai:
    transforms:
      - {include: "vllm:time_to_first_token_seconds", action: update, new_name: gen_ai.server.time_to_first_token}
      - {include: "vllm:inter_token_latency_seconds", action: update, new_name: gen_ai.server.time_per_output_token}
      - {include: "vllm:e2e_request_latency_seconds", action: update, new_name: gen_ai.server.request.duration}

As a result, the service name of the span became vllm , the token count attribute was changed to input_tokens, output_tokens (the old name was removed), and three metric names were changed to gen_ai.server.* . However, just from the names, it is not possible to know whether the meaning of the values also matches the standard. For example, the standard's time to first token is defined as "until the first token of a successful response," and it was not verified whether vLLM's value covers the exact same interval (unmeasured). Additionally, as seen earlier, units are empty and must be filled in separately.

5. How to Find Slow Requests

5.1 Isolating Time to First Token Using Only Span Attributes

Using 500 request spans collected from four servers (two with tracing on, two with verbose on), we compared 69 requests where the first token took 1 second or more with 38 requests where it took less than 0.2 seconds.

The median proportion of queueing time in slow requests' TTFT was 59%. Slow requests were slow because of waiting in line, not because computation was slow, which points not to "getting a faster GPU" but to "increasing concurrency limits and replica counts or rate-limiting requests." The time to first token was not exactly equal to the sum of queueing and prefill; the median difference was 78ms, reaching up to 221ms (presumed to be tokenization or the initial decode phase, but not broken down). The total request duration was equal to the sum of the time to first token and decode time within 1.1ms.

5.2 Tail Sampling vs. Probabilistic Sampling

We sent 120 requests under a mixed workload (input lengths of 64, 512, and 1,500 tokens, concurrency of 16). The client-measured request latency had a median of 2.08s, p90 of 2.73s, and maximum of 3.02s, with 19 requests having a server-side span duration of 2.5 seconds or more.

• Tail sampling determines what to keep by evaluating latency after the trace finishes, so it kept all slow requests. The minimum latency of the remaining traces was 2,551ms, and only 16% of the total traces were retained.
• Probabilistic sampling selects traces without knowing if they are slow, so slow requests were retained only in proportion to the sample rate. Although 10% was sampled, 13% actually remained due to hash variance, and only 1 of them was slow. While the average latency might look similar, the tail you most want to see disappears.
• The price of tail samplingis that the same collector must gather the entire trace and hold it in decision_wait memory until then. Because this test had only 120–300 traces and the cost wasn't obvious, we measured the same collector (0.162.0) and policy separately on a Mac laptop (Apple M1 Max) using a synthetic workload (2 spans and 12 attributes per trace). To avoid losing data before making a decision, if you set num_traces to "traces per second × decision_wait × 1.2", the Go heap increased by about 4KB (2.6–4.7KB) per trace held in memory. At 10,000 spans/sec with decision_wait at 5s, 10s, and 30s, the heap was 161MB, 293MB, and 850MB (RSS 385MB, 546MB, 1.2GB, where 30s was from a run with less CPU contention); at 43,000–45,000 spans/sec with 10s and 30s, it was 1.2GB and 3.3GB (RSS 1.7GB, 4.0GB). The RSS of the same pipeline without tail sampling was 268–292MB at the same throughput. Memory is ultimately proportional to "span rate × decision_wait".
• num_traces If you set too small (10,000) under heavy load, it will silently drop traces. When the map fills up, it deletes the oldest traces first, and traces deleted before a decision is made are not retained. At 1,000 spans/sec nothing was lost, but at 10,000 spans/sec, 88.5–88.7% of new traces disappeared before a decision, and at 39,000–47,000 spans/sec, 97.3–97.6% disappeared, leaving the 10% probability policy with actual retention rates of 1.2% and 0.3%. There were no warnings in the collector logs, and it was only visible in the otelcol_processor_tail_sampling_sampling_trace_dropped_too_early metrics. num_traces It is safer to set
• to at least greater than "traces per second × decision_wait" and set up alerts on this metric.

For traces spanning multiple services, they must not go to different collectors; they must be routed to the same place using the trace ID (as explained in the official documentation, though not verified in this test).

• tail_sampling 6. Issues Encountered numeric_attribute The policy kept nothing for this attribute. Our first attempt filtered by the minimum value of the time-to-first-token attribute (float, in seconds), resulting in 0 remaining traces with no warnings or errors. We could not determine the cause (the documentation example uses an integer value, and min_value also appears to be an integer). It worked after changing to the latency policy, which checks trace length. It is dangerous that the policy remains silent when it drops everything. When changing sampling configurations, you must always count the remaining items.
• metricstransform The name is no longer recommended. When starting up, the collector warned that "the alias is deprecated, use metrics_transform instead".
• Truncating a file that a collector currently has open fills the beginning of the file with NUL bytes. When we emptied the export file with : > file , the collector continued appending to its previous position, filling the beginning with \0 and causing JSON parsing to fail on the first line. To empty a file safely, it is better to stop the collector or use a new path.
• The Prometheus receiver logs scrape failure warnings every 5 seconds before the server starts up. These disappeared once the server was ready, and there was no impact on the data.
• The issue where the vLLM server's KV cache allocation was smallest on the first launched server (Part 14) also occurred in this test. The first launched server (22,048 tokens) differed from subsequent servers (26,560 tokens), and only the subsequent servers were used for overhead comparisons.

7. OTCA Scope and Further Reading

This summarizes which items of the test scope are directly demonstrated by this experiment.

The LabHub OTCA course includes an additional module (otca-llm-serving, Lab Step 7) where you can calculate these contents yourself with actual spans and histograms. It is visible in the deployed environment.

8. Limitations and Unmeasured Items

• One server, one model (1.7B), and one GPU. How traces connect across multiple servers (between gateways, routers, and workers in disaggregated serving) was not measured (Unmeasured).
• ). Tracing in SGLang and OTel export in Dynamo were not measured (Unmeasured). vLLM values are not generalized to other engines.
• Trace and metric storage backends were not attached. Visualization, alerting, and storage costs are Unmeasured .
• Settings to record prompt and response contents as spans or events, and the resulting privacy issues, were not covered.
• The overhead of the collector itself was measured separately on a Mac laptop (Apple M1 Max) using the same collector (0.162.0) and synthetic load. CPU usage was almost proportional to span throughput: 0.13–0.15 cores per 10,000 spans per second (receiving and writing to file only), 0.14–0.16 cores with the transform processor added, and 0.57–0.63 cores at 45,000–47,000 spans per second with an RSS of about 290MB. RSS was 186–199MB even when mostly idle. Receiver rejections and export failures were 0 under all conditions. For latency, collector processing itself had a median of 3.5ms, and the rest was determined by configuration. With batching of 1 second and file exporter's 1-second flush like in this article, it took 0.94–1.99 seconds (median 1.46 seconds) for spans to appear in the file, and 6.0–7.0 seconds when tail sampling was added. Since these values were measured on a Mac laptop where other experiments were running simultaneously and the CPU was mostly saturated, they may differ from server CPU values. It is better to trust the trends—"CPU usage is proportional to throughput" and "latency is determined by batching and decision_wait"—rather than the absolute values. Network and retry costs when sending to remote storage via OTLP, and the rate limit where the collector starts rejecting receives, were not measured ( batch Unmeasured).).
• Overhead values were measured by launching the server twice, and standard deviations were not calculated. While the direction of the conclusion that it is within 1% is certain, it is best not to trust digits below the decimal point.
• Since GenAI conventions are still in development, the tables in this article are based on 2026-10-06 and are subject to change soon.

References

• OpenTelemetry GenAI Semantic Conventions (Repository): https://github.com/open-telemetry/semantic-conventions-genai (Inference spans docs/gen-ai/client-inference.md, Server metrics docs/gen-ai/gen-ai-metrics.md)
• OpenTelemetry Collector contrib 0.162.0 and Tail Sampling Processor Documentation: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.162.0/processor/tailsamplingprocessor
• vLLM Tracing: vllm serve --help=all 's --otlp-traces-endpoint, --collect-detailed-traces, and the installed version's vllm/tracing/ code
• Parts 14 (vLLM Load and KV Cache), 24 (Dynamo), and 20 (Decomposition of Real-time Audio Latency) of this series