When documentation still calls a metric name that vLLM deleted: How an llm-d docs PR was merged in 2 hours and 9 minutes

Result: Merged (2026-10-17 03:08, llm-d/llm-d#2835). In fact, this was taken from GitHub PR, review, and check records, llm-d source (the base commit of the PR 696b5e81), vLLM source (release tag), llm-d autoscaler source (tag v0.9.0), and raw measurement data we left behind. Times are in KST. As of 2026-10-17, no llm-d release includes this change yet (latest is v0.10.0, 2026-09-29).

Table of Contents

• One-line Summary
• 1. What was the problem: Symptoms we observed
• 2. Cause: Which line in the source code
  • Engine side: The name disappeared as announced
  • Consumer side: Code followed, only documentation remained
• 3. What was changed: Summary of diff
• 4. Reaction: 2 hours and 9 minutes without a bot or inline comments
• 5. What we learned: Tips for open-source contribution
• Links

One-line Summary

1. What was the problem: Symptoms we observed

First, to be honest, llm-d does not exist in the TT Lab cluster. We encountered this issue not as an operational incident, but during our studies. While reading the code of projects surrounding the vLLM engine (production-stack, LMCache, llm-d), we traced "how metric names defined by the engine propagate to queries in other repositories," and the details were summarized in "Reading Open Source Architecture" Part 4.

To avoid comparing names solely based on documentation text, right before submitting the PR, we spun up vLLM v0.31.0 (vllm/vllm-openai image, latest release at the time) as a disposable pod on a single GPU node (8GB class GPU) in a small cluster and pulled /metrics . To avoid downloading anything new, we uploaded a random-weight model with only configuration files created (Qwen2 structure, 2 layers, hidden 256, --load-format dummy) and sent 5 requests (generating 16 tokens each) where the first 160 tokens matched. The output did not contain the names called by the llm-d documentation.

The raw data contained 0 lines starting with the two old names, and the values that appeared instead were as follows.

vllm:prefix_cache_queries_total{engine="0",model_name="tiny"} 805.0
vllm:prefix_cache_hits_total{engine="0",model_name="tiny"} 640.0
vllm:inter_token_latency_seconds_count{engine="0",model_name="tiny"} 75.0

The numbers match the request shape. 5 requests × 161 tokens = 805 lookups, where the first request misses all tokens and the remaining four hit 160 tokens each, totaling 640 (block size 16, same output cache_config_info). The token interval for 5 requests generating 16 tokens each is 15 × 5 = 75. The hit rate is 640 / 805 ≈ 0.795.

PromQL written with non-existent names returns empty results without raising errors (standard Prometheus behavior). Readers who follow the guide and run the queries will only see "nothing comes out" and have to figure out the reason themselves. The dashboard panels are also expected to appear empty, though we did not upload them to Grafana (unmeasured).

The results of scanning all documentation and guides in the llm-d v0.10.0 tag using the same method are also in Part 4. vllm: Out of 71 names, 40 were in the output, 27 were conditional metrics that appeared only when offload or NIXL connectors were enabled, and 4 had no definition in the v0.31.0 source. Three of those four belong to the time_per_output_token_seconds family. Since the PR was based on the main branch, the fairness guide (which is not yet in v0.10.0) was added.

2. Cause: Which line in the source code

llm-d line numbers are based on the PR's base commit 696b5e81 , and vLLM is based on the release tag.

Engine side: The name disappeared as announced

vLLM has a rule when removing metrics. According to docs/usage/metrics.md lines 78–82 , deprecation is announced in X.Y, hidden in X.Y+1 (can be revived with--show-hidden-metrics-for-version=X.Y ), and deleted in X.Y+2.

• vllm:gpu_prefix_cache_hit_rate is a V0 engine gauge. In v0.9.0's vllm/engine/metrics.py lines 119–128 , it is in a hidden state with comments stating "Deprecated in 0.8, replaced by lookup and hit counters in V1" and "Hidden in 0.9, scheduled for deletion in 0.10", and the definition is absent in the same file of v0.10.0. The V1 engine emits two counters instead of hit rate: vllm:prefix_cache_queries·vllm:prefix_cache_hits . The design document (docs/design/metrics.md lines 437–447) explains the rationale: "Instead of fixing the ratio with a gauge, let users derive the ratio over desired intervals like rate(…[5m]) ." Notably, the baseline environment at the head of the fairness guide is vLLM 0.26 . Even in the version where the guide was verified, this gauge was already gone.
• vllm:time_per_output_token_seconds is a V1 engine histogram. In v0.11.0's vllm/v1/metrics/loggers.py lines 387–388 notes "deprecated in 0.11, vllm:inter_token_latency_seconds renamed to," and v0.12.0 (lines 689–692) changes it to hidden with a TODO stating "removed in 0.13.0." In reality, it remained hidden until v0.14.0 (lines 718–721), and the definition is absent from the same file in v0.15.0 (released 2026-01-29). The current name is vllm:inter_token_latency_seconds .

The engine retired names according to the rules, and sometimes more leniently. The problem is that the consumers of those names reside in a different repository. Additionally, the counter name in the source does not have _total , but since prometheus_client adds it when exporting, queries must be written as vllm:prefix_cache_hits_total . This is a frequent point of confusion when grepping the source to migrate names.

Consumer Side: Code Followed, Only Documentation Remained

• The autoscaler design document docs/architecture/autoscaling/wva.md still listed the old name in the metrics list on line 144 and in the line 210 query. However, line 70 of avg_itl in the collector code described by this document (repository named llm-d-autoscaling) v0.9.0 internal/collector/registration/queueing_model.go line 70 is already querying vllm:inter_token_latency_seconds_sum·_count . The documentation diverged from the code.
• In the DigitalOcean dashboard JSON, the "Time Per Output Token Latency" panel had its p95, p90, and p50 queries (line 1544, line 1559, line 1575) all passing vllm:time_per_output_token_seconds_bucket into histogram_quantile .
• The observation table in the fairness guide line 663 lists as the "mechanism optimized by the strategy" vllm:gpu_prefix_cache_hit_rate to look at.

3. What was changed: Summary of diff

Only names were changed. Sentences, panel titles, and the rest of the query structures were left as they were.

 # docs/architecture/autoscaling/wva.md
-- `vllm:time_per_output_token_seconds` -- ITL histogram (sum/count for average)
+- `vllm:inter_token_latency_seconds` -- ITL histogram (sum/count for average)
-| `avg_itl` | `rate(vllm:time_per_output_token_seconds_sum[1m]) / rate(..._count[1m])` | ...
+| `avg_itl` | `rate(vllm:inter_token_latency_seconds_sum[1m]) / rate(..._count[1m])` | ...

 # docs/infrastructure/providers/digitalocean/monitoring/inference-gateway-dashboard.json (3곳)
-"expr": "histogram_quantile(0.95, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[$__rate_interval])))",
+"expr": "histogram_quantile(0.95, sum by(le) (rate(vllm:inter_token_latency_seconds_bucket[$__rate_interval])))",

 # guides/turn-priority-fairness/README.md
-| `vllm:gpu_prefix_cache_hit_rate` | The mechanism the strategy is optimizing. |
+| `rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m])` | Prefix-cache hit ratio: the mechanism the strategy is optimizing. |

In place of the hit rate, we put a ratio expression of two counters instead of a single gauge name. This is because there is no gauge with the same name to replace it, and it is also the shape recommended by the vLLM design document. The section [5m] was left identical to the example in the design document.

In the PR body, we put the files, old names, new names, and reasons in a single table, and attached the three lines of grep output and 「0 lines of old names」 from section 1 as they were. We also noted that 「the dashboard JSON still parses, we did not upload and test it in Grafana, only the names within the histogram_quantile three expressions were changed while panel titles remain unchanged」. The CONTRIBUTING guide of llm-d states that AI-assisted contributions are accepted under the same criteria as other contributions, but the submitter must be able to explain the content directly as the author of record. Therefore, we added a sentence at the end of the body stating that it was written with AI assistance and personally reviewed. There is a single commit, signed with the repository rules DCO sign-off (Signed-off-by) and an SSH signature. The checks also include an item to verify if the commit is signed.

4. Response: 2 hours and 9 minutes with no bots or inline comments

Passed checks include DCO, signed commit verification, pre-commit, CI Guides Check, guide dry-run, and site preview build. The 9 skipped checks were tasks like image builds, which appear to have been skipped because it was a documentation-only PR.

Unlike the SeaweedFS in Part 1, no review bot attached, and unlike the Grafana charts in Part 2, there wasn't even a single word of 「Thanks!」. It ended with just two approvals and no inline comments or questions. There was nothing for us to handle separately. The author of the merge commit 2b0b8de7 is our account, and the message is our commit message title appended with the PR number.

There is one more PR with the same shape. Since the default dashboard of production-stack also calls the disappeared vllm:gpu_prefix_cache_* name, vllm-project/production-stack#1147 was submitted and is in progress as of 2026-10-17.

5. What we learned: Tips for open source contribution

• A single name spreads across multiple repositories. The engine handled deprecation, hiding, and deletion according to rules, but those rules are only kept within the engine repository. Guides, design documents, and dashboard JSONs reside in other repositories, and wrong names do not cause failures—they simply result in emptiness. A single /metrics engine search and a single list of consumer vllm: names can help find such places.
• Do not guess alternative names; find them in the code. The alternative name in the WVA document was already being used by the collector code described by that document. By showing 「the documentation differs from the code」 with lines of code, the maintainer has almost nothing extra to verify.
• Watch out for similar names. v0.31.0 had vllm:request_time_per_output_token_seconds there is another histogram called . From the same raw data, time_per_output_token_seconds without anchoring the line start with grep yields 27 lines, while anchoring it with ^vllm:time_per_output_token_seconds yields 0 lines. Furthermore, the source descriptions differ: one is "latency between tokens" and the other is "time per output token for a request". Saying "their names are similar, so let's change it to this" is a recipe for a wrong PR.
• It treats measured and unmeasured things with equal weight. The sentence saying it wasn't uploaded to Grafana didn't weaken the PR. Maintainers judge based on what has been verified and what has not.
• Documentation PRs also have verification paths. llm-d builds the site and provides a preview even when only documentation changes, and it runs guide checks and dry-runs. The smaller the PR, the more that all these checks being green means the reviewer only has to look at a few lines of diff.

A word from Titi. A friend changed their name, but the old name remained in the address book. Since they wouldn't answer when called, everyone just said, "We can't reach them." After calling them directly (0 lines!), confirming the new name (75!), and updating three places in the address book, a "thumbs up" stamp was given in just two hours. I decided to be careful not to accidentally call another friend with a similar name.

Links

All materials were accessed on 2026-10-17.

• PR: llm-d/llm-d#2835 (Merge commit 2b0b8de7)
• llm-d source (base commit): docs/architecture/autoscaling/wva.md, inference-gateway-dashboard.json, guides/turn-priority-fairness/README.md, CONTRIBUTING.md
• Autoscaler collector: queueing_model.go(v0.9.0)
• vLLM metrics deprecation policy: docs/usage/metrics.md(v0.31.0), Reason for choosing the counter: docs/design/metrics.md(v0.31.0)
• vLLM metric definitions: vllm/v1/metrics/loggers.py(v0.31.0), Old V0 gauge vllm/engine/metrics.py(v0.9.0)
• Similar-shaped PR (in progress): vllm-project/production-stack#1147