vLLM production-stack · LMCache · llm-d: Code map of the engine frontend and KV cache tier

Status: Draft (2026-10-17), partially measured. Reference versions: production-stack vllm-stack-0.1.13 (commit e8cb4959ebfa), vLLM v0.31.0 (commit db9527a46873), LMCache v0.5.6 (commit 3e11e10812b4), llm-d v0.10.0 (commit f22bbaf3173a), llm-d-router v0.11.0 (commit a5cbe600ebad). Since a small cluster does not keep vLLM running constantly, no new pods were spawned today; measurements cite raw data collected as a disposable pod on a single GPU node (8GB laptop GPU) on 2026-10-16, recounted on 2026-10-17. Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. One-line Summary: One Kitchen, Interchangeable Floor Manager and Refrigerator
• 2. Architecture Diagram
• 3. Code Map
• 4. Following the Core Paths
  • 4.1 production-stack: From a request to the engine
  • 4.2 vLLM ↔ LMCache: The Connector Interface
  • 4.3 llm-d: How EPP selects a pod
  • 4.4 Metric Path: A single definition becomes queries in three places
• 5. Connection to User Contributions
• 6. Measurements
  • 6.1 What and how was measured
  • 6.2 Counting /metrics output lines
  • 6.3 Are consumer-requested names present in the output?
  • 6.4 Disappeared flags
• 7. Where to start reading
• 8. Good places for your first contribution
• 9. Limitations
• References

1. One-line Summary: One Kitchen, Interchangeable Floor Manager and Refrigerator

vLLM is a kitchenthat runs models fast on a single GPU (or a few). When operating multiple kitchens, you need a floor managerto decide which kitchen receives an order, a refrigeratorto share prep work (KV cache) across isolated kitchens, and an order board (metrics) to report how busy the kitchens are. These three projects keep the kitchen (vLLM) intact while filling its surrounding environment in different ways.

Why KV cache is needed and what LMCache saves are covered in the Anatomy of LLM KV Cache series. Instead of principles, this installment focuses on which function in which file does the work.

2. Architecture Diagram

Here is how to read the diagram.

• The frontend is one of two. The production-stack router on the left directly receives the request body and proxies it to the engine. In llm-d on the right, a proxy like Envoy holds the body, and EPP uses an ext-proc callback to return only the address instructing "send to this pod".
• Both frontends periodically read the engine's /metrics to understand pod busyness. Additionally, the llm-d EPP subscribes to KV events (dotted lines) pushed via ZMQ by the engine to know in real time "which pod has which block".
• Inside the engine, 16-token GPU blocks serve as the primary cache. If they are insufficient, the KV connector calls LMCache, which searches through the tiers sequentially from the front.
• In MP mode, where LMCache runs as a separate server outside the engine, the bottom-most box becomes an entirely distinct process.

3. Code Map

First, let's organize "where the code resides in repositories." The llm-d/llm-d repository contains no service code. Among 1,315 files in the v0.10.0 tag, 834 are YAML, 249 are Markdown, 0 are Go files, and Python files consist of only 33 CI, tool, and example scripts. It is a collection of deployment guides, Helm values, and design documents, while the actual components reside in separate repositories pointed to by the Roster table in COMPONENTS.md . EPP is llm-d-router . Its former name llm-d-inference-scheduler address now redirects to llm-d-router (HTTP 301 verified on 2026-10-17). Since the guides/env.sh of llm-d v0.10.0 pins the EPP image v0.11.0 and the Gateway API Inference Extension (GAIE) v1.5.0 CRD, this post reads llm-d-router v0.11.0 fetched separately. API types such as InferencePool are imported from the GAIE module.

4. Following the Core Path

4.1 production-stack: From Request to Engine

1. Upon startup, the router sequentially instantiates service discovery, engine statistics collector, request statistics, and routing logic as singletons in initialize_all . This order is necessary because the routing logic reads the tables generated by the preceding three components.
2. The engine list is created by K8sPodIPServiceDiscovery._watch_engines . It uses a label selector to watch pods, and queries _get_model_names for /v1/models on each ready pod to record which IP serves which model. It connects directly to pod IPs rather than a Service because the router must select a specific pod, whereas a Service delegates that choice to kube-proxy.
3. Requests pass through route_chat_completion and enter route_general_request . Here, candidate engines are filtered using model from the text, a URL is obtained by calling route_request in the routing logic, and the response is streamed via process_request . A retry loop that removes the failed URL and selects another is also contained within this function.
4. The routing logic consists of eight types in the RoutingLogic enum. The primary axes are RoundRobinRouter, which places session keys on a hash ring, SessionRouter, which hashes prompt strings into 128-character chunks and inserts them into a trie, and PrefixAwareRouter(HashTrie, which queries the LMCache controller for the instance with the longest matching token prefix KvawareRouter .

The difference between the two prefix routers highlights the project's design considerations. PrefixAwareRouter infers state solely based on records of what it has sent. As the class comment explicitly states, "assuming no prefix cache eviction," the router remains unaware even if the engine deletes blocks. However, it has no external dependencies. KvawareRouter queries the LMCache controller for the actual stored location, making it accurate, but it requires the LMCache package to be included in the router image. The default router image embeds pyproject.toml 's optional dependency lmcache==0.3.11 , which differs significantly from the engine side, and whether the controller messages between the two match remains unverified (unconfirmed).

The Helm chart exposes these router arguments directly. deployment-router.yaml passes routerSpec.routingLogic to --routing-logic , and deployment-vllm-multi.yaml assembles the modelSpec command for each vllm serve . When LMCache is enabled, --kv-transfer-config receives LMCacheConnectorV1 , and the size is provided via the LMCACHE_* environment variable. Because engine arguments are assembled as strings, if vLLM deprecates a flag, the chart and tutorial quietly become outdated and break at runtime. Section 5 addresses contributions in this area.

4.2 vLLM ↔ LMCache: The Connector Interface

vLLM v1 abstracts external KV storage into a single KVConnectorBase_V1 . The key point is that the same object is instantiated for two roles.

• Scheduler side: get_num_new_matched_tokens answers "the number of tokens not in the GPU but fetchable from outside." The scheduler checks this value inside schedule (line949), decides to skip computation for those tokens, and uses update_state_after_alloc and build_connector_meta to create metadata for what to load and what to save.
• Worker side: Receives the metadata, initiates loading via start_load_kv , calls wait_for_layer_load·save_kv_layer layer by layer, and finalizes saving with wait_for_save at the end.

Why are they split? The scheduler schedules batches without touching GPU tensors, whereas the worker directly handles GPU tensors, and the two may even run in separate processes. Because "how much can be loaded" must be known before scheduling batches, while actual copying must overlap with forward passes, the architecture connects decision-making and execution through a single block of metadata. The per-layer function aims to overlap copy time by loading the KV of the next layer while computing the current one.

The LMCacheConnectorV1 inside the vLLM tree is a thin shell.

use_native = vllm_config.kv_transfer_config.get_from_extra_config(
    "use_native", False
)
if use_native:
    from vllm.distributed.kv_transfer.kv_connector.v1 import lmcache_integration
    _adapter = lmcache_integration.vllm_v1_adapter
    cls = _adapter.LMCacheConnectorV1Impl
else:
    from lmcache.integration.vllm.vllm_v1_adapter import (
        LMCacheConnectorV1Impl as LMCacheConnectorLatestImpl,
    )
    cls = LMCacheConnectorLatestImpl

self._lmcache_engine = cls(vllm_config, role, self)

Source: vllm v0.31.0 vllm/distributed/kv_transfer/kv_connector/v1/lmcache_connector.py(lines 88–108, log lines omitted), Apache-2.0

The default value is the implementation in the installed LMCache packageIf the implementation were locked inside vLLM, LMCache would have to wait for a vLLM release every time a fix was made. Therefore, vLLM provides the interface while LMCache owns the implementation. The MP connector also lmcache_mp_connector.py looks for the LMCache implementation first, and uses the copy in the tree only when it is not found.

The actual implementation on the LMCache side is LMCacheConnectorV1Impl . Lookups start from get_num_new_matched_tokens , go through the lookup client, and tokens are ChunkedTokenDatabase by _chunk_tokens and split by _prefix_hash , chaining the previous chunk hash to generate the key. The default chunk size is config.py tokens, which is 16 times larger than a vLLM GPU block (measured at 16 tokens). This is because larger chunks reduce the number of objects and round trips in the remote storage, at the cost that overlaps shorter than 256 tokens will not be cached in LMCache.

The storage tier is where CreateStorageBackends puts OrderedDict into The orderis the lookup order (among PD, LocalCPU, P2P, NIXL, LocalDisk, GDS, Maru, and Remote, whichever are enabled). StorageManager.batched_get It searches from the front, and anything found remotely is written back to LocalCPU. The remote side storage_backend/connector/ branches out into seventeen adapters (Redis, Valkey, S3, file system, Mooncake, etc.).

MP mode takes this entire layer outside the engine. MPCacheServer assembles multiple engine modules, and MPServerConfig uses ZMQ (port 5555) for its default transport, 256 for chunks, and blake3 for hashing. The goal is to allow the cache to persist even if the engine restarts and to be shared across multiple engines.

4.3 llm-d: How EPP chooses pods

The llm-d Router is divided into a proxy and an EPP. When the proxy intercepts a request and asks the EPP via an ext-proc stream, the StreamingServer.Process of the EPP receives it, runs the scheduler, and the generateHeaders puts the chosen pod address in the x-gateway-destination-endpoint header and sends it back. It is a decoupled architecture that leaves body transmission, TLS, and retries to a proven proxy while isolating LLM-specific decisions, which is the exact opposite of a production-stack router that handles both in a single Python process.

The scheduler Scheduler.Schedule is a loop of "running the profile chosen by the profile handler, checking the results, and choosing again." A single profile goes through SchedulerProfile.Run inside: filter → weighted scorer (runScorerPlugins) → picker. The reason profiles can be run multiple times is P/D separation. The disagg handler's Pick first selects a D pod using the decode profile, and if necessary, selects a P pod using the prefill profile. The actual P→D flow is handled by the routing sidecar (llm-d-router pkg/sidecar/proxy/) next to the decode pod, which sends the request to P max_tokens=1 first, attaches the received KV transfer information, and passes it to D. The principles of P/D separation itself are covered in the NVIDIA Dynamo post.

The numbers read by the scorer come in through two channels. One is the datalayer source that scrapes the engine /metrics , where parseMetrics parses the Prometheus text and determines which engine-specific label maps to which attribute.

var defaultEngineConfigs = []engineConfigParams{
	{
		Name:                "vllm",
		QueuedRequestsSpec:  "vllm:num_requests_waiting",
		RunningRequestsSpec: "vllm:num_requests_running",
		KVUsageSpec:         "vllm:kv_cache_usage_perc",
		LoRASpec:            "vllm:lora_requests_info",
		CacheInfoSpec:       "vllm:cache_config_info",
	},

Source: llm-d-router v0.11.0 pkg/epp/framework/plugins/datalayer/extractor/metrics/factories.go(lines 91–99), Apache-2.0

The same table lists SGLang, TensorRT-LLM, and Triton side by side, and selects the table using the pod label llm-d.ai/engine-type . This design ensures that even if an engine changes its name, only this single table needs to be modified. Taking these values, KVCacheUtilizationScorer.Score gives 1 − KV 사용률 as a score.

The other is KV events. When vLLM stores or evicts blocks, it publishes events like BlockStored to ZmqEventPublisher , and EPP's precise-prefix-cache plugin opens a subscription for each pod via ensureSubscriber . The received messages are unpacked by VLLMAdapter.ParseMessage to add or remove from the block index. This replaces the "guessing from sent history" in PrefixAwareRouter with facts reported by the engine, and in return, --kv-events-config and open the ZMQ port.

4.4 Metric Path: A Single Line of Definition Becomes Queries in Three Places

PS in the diagram stands for production-stack. The starting point is the PrometheusStatLogger constructor, where vllm: metrics are defined line by line. The KV cache usage is vllm:kv_cache_usage_perc, the prefix cache is vllm:prefix_cache_queries·vllm:prefix_cache_hits, and the inter-token latency is vllm:inter_token_latency_seconds . The source counter names do not have _total , but when prometheus_client exports them, _total and _created is attached, so the querying side must read it as vllm:prefix_cache_hits_total . This difference is one of the causes of name confusion. This registry attaches attach_router to the /metrics path.

vLLM exposes prefix caching as "two lookup and hit counters" instead of a "hit rate gauge." This is so users can calculate the ratio for the desired interval like rate(hits[5m]) / rate(queries[5m]) (docs/design/metrics.md). When removing a metric, it follows the rule of X.Y deprecation notice, X.Y+1 hiding (restored as--show-hidden-metrics-for-version ), and X.Y+2 deletion (docs/usage/metrics.md). The engine-side rule is clear, but the problem is that the side reading this name is in a different repository.

First, the consumer, the production-stack router collector EngineStatsScraper , polls every 30 seconds by default from each engine /metrics is GETted and from_vllm_scrape extracted by matching the name precisely.

for family in text_string_to_metric_families(vllm_scrape):
    for sample in family.samples:
        if sample.name == "vllm:num_requests_running":
            num_running_reqs = sample.value
        elif sample.name == "vllm:num_requests_waiting":
            num_queuing_reqs = sample.value
        elif sample.name == "vllm:gpu_prefix_cache_hit_rate":
            gpu_prefix_cache_hit_rate = sample.value
        elif sample.name == "vllm:gpu_prefix_cache_hits_total":
            gpu_prefix_cache_hits_total = sample.value
        elif sample.name == "vllm:gpu_prefix_cache_queries_total":
            gpu_prefix_cache_queries_total = sample.value
        elif sample.name == "vllm:gpu_cache_usage_perc":
            gpu_cache_usage_perc = sample.value

Source: production-stack vllm-stack-0.1.13 src/vllm_router/stats/engine_stats.py(lines 63–76), Apache-2.0

If there is no matching name, 0 remains without any exception or log. The router exposes this value back as a gauge ( metrics_router.py of /metrics with a prefix like vllm: prefixattached (server label). Therefore, the same name gets mixed from both the engine and router sources with different labels, and the default dashboard's prefix cache panel filters by endpoint="service-port" . service-port is the name the chart assigns to the engine Service port, and the router Service port name is router-sport , so it keeps only the time series coming from the engine.

The second consumer is Prometheus and Grafana. The chart's ServiceMonitor scrapes the engine and router, and the PromQL in the dashboard JSON references these names as strings. The third consumer is the EPP label from Section 4.3. If the names are wrong in any of the three, it will silently result in 0 or "No data" rather than failing.

5. Connection to User Contributions

The author's four PRs all share the same pattern: "another repository still calls a name that vLLM deleted" (statuses as of 2026-10-17). In this post, we reopened only the v0.31.0 tag to verify "it is gone now, and this is here instead."

#1147 only fixed the dashboard, and the same issue in the router parser (engine_stats.py:69–76) is handled separately by #1140 and #1146 as mentioned in the PR body. Since llm-d#2835 is based on main, in the v0.10.0 tag, the same sentence is located near docs/architecture/advanced/autoscaling/wva.md:144·210줄 and the DigitalOcean dashboard 1544줄 , and the third fixed guide (turn-priority-fairness) is not yet in v0.10.0. The reason for the different paths was not verified (unverified).

6. Measurement

6.1 What and How We Measured

Measured on 2026-10-16 by running a disposable pod on a single GPU node (8GB-class laptop GPU) in a small cluster. The measurement script (pod.sh) fixes the node and uses only existing images (imagePullPolicy: Never), blocks Hugging Face access, and sets a memory limit of 3.5Gi and a 90-minute timeout. The model is a randomly weighted Qwen2 architecture(2 layers, hidden 256, --load-format dummy). The image is vllm/vllm-openai v0.31.0 (PyTorch 2.13, CUDA 13.0). After sending 5 requests with the same first 160 tokens, we received /metrics . Since the router, LMCache, and EPP were not launched, Section 6.3 is a static comparisonmatching the names extracted from each tag source with this output.

6.2 /metrics Output Count

The numbers match up. 5 requests × 161 tokens = 805 lookups; the first request misses all, and the remaining four hit 160 tokens (10 blocks) each, totaling 640. The token interval for 5 requests created with 16 tokens each is 15 × 5 = 75. 144,272 and 272 blocks × 16 = 2,308,352 is also the same as in the kv_cache_size_tokens line (the model is very small, so there are many blocks).

6.3 Are Consumer-Facing Names Present in the Output?

The four "not defined" items in llm-d documentation are time_per_output_token_seconds three of the family (fixed by #2835) and vllm:kv_offload_tiering_block_hits_total one. The last one was left as a contribution candidate in Section 8.

6.4 Missing flags

The chart default engine image read on the same day (lmcache/vllm-openai:latest) was vLLM 0.29.0 and LMCache 0.5.5, and even inside it, loggers.py there was no gpu_prefix_cache . Even with the chart defaults as they are the router parser and prefix cache panels will show 0 or No data. Grafana was not actually installed and viewed with the chart (unmeasured).

7. Where to read first

1. vllm/v1/metrics/loggers.py 's PrometheusStatLogger constructor is scanned from top to bottom. What the engine counts is the input for everything else.
2. vllm_router/app.py 's initialize_all starting from request.py 's route_general_request to follow a single request all the way through.
3. routing_logic.py Compare RoutingInterface and a couple of implementations. Looking at the tests (src/tests/test_*_router.py) together helps quickly grasp the expected behavior.
4. kv_connector/v1/base.py After reading the top comments (scheduler/worker function list), jump to the LMCache vllm_v1_adapter.py function of the same name.
5. llm-d standardizes terminology with docs/architecture/core/router/README.md , and for code, look at llm-d-router scheduler.go and factories.go .

8. Good first contribution opportunities

These are things I found while reading the source code. I haven't checked if there are already open PRs or issues, so you should search before submitting.

1. production-stack per-cloud dashboard copies. The Helm chart dashboards are being fixed in #989 and #1147, but tutorials/terraform/{coreweave,eks,nebius}/config/ 's vllm-dashboard*.json four sets still have vllm:gpu_cache_usage_perc and vllm:gpu_prefix_cache_* remaining in 10 places each (e.g., eks/config/vllm-dashboard.json:732).
2. in the production-stack benchmark tutorial, --swap-space 0. 07-benchmark-multi-round-qa-single-gpu.md:51 and three other tutorial sets in 07 and 08 give extraArgs: ["--swap-space", 0] . v0.31.0 exits with code 2 on this flag (Section 6.4).
3. Metric names in the llm-d P2P guide. guides/p2p-kv-cache-sharing/benchmarking/README.md:129 uses vllm:kv_offload_tiering_block_hits_total , but v0.31.0 only defines vllm:kv_offload_tiering_chunk_hits . We first need to check whether the name was changed in the vLLM version the guide relies on (after vllm#48798).
4. KvawareRouter chat request in route_request . request_json.get("prompt", "") only tokenized and left TODO: Handle chat completions behind. /v1/chat/completions requests seem to be queried with an empty prompt, falling back to session and QPS routing. You can refer to the handling of PrefixAwareRouter in the same file, messages .
5. Mixed time series in the default dashboard (unverified). The "Number of Running Requests" panel plots vllm:num_requests_running without a label. Since the router also emits the same name (Section 4.4), the two time series may be plotted together. You need to check the actual screen first.

Project Guidelines: production-stack requires a proposal in proposals/ for major changes, and a DCO Signed-off-by for commits. LMCache has devas its default branch, requires DCO, has a collection of beginner-friendly issues (LMCache#627), and includes an "AI-generated code signals" review item in its AGENTS.md and coding standards. llm-d accepts AI-assisted contributions under similar standards, provided the submitter can personally explain the content. All of the author's PRs disclosed AI assistance at the end of the body.

9. Limitations

• There are no measurements where the router, LMCache, and EPP were actually running and routing live requests (unmeasured). The comparison in Section 6.3 is a static comparison matching strings in tag sources against engine outputs.
• The measured engine uses a single very small model with random weights and a single GPU. We did not examine metrics that only appear when LoRA, speculative decoding, or KV connectors are enabled, nor multi-process metrics modes for multiple API servers.
• We did not cover production-stack's Go operator, experimental features like semantic cache and PII, LMCache's CacheBlend and controller internals, nor llm-d's autoscaler (WVA) and batch gateway.
• For when old names and flags were hidden or removed across versions, we relied on the investigations in PR bodies without opening tags prior to v0.31.0.
• Performance figures (TTFT and throughput) are out of scope for this post. LMCache measurements are in the LMCache post, and vLLM load experiments are in vLLM Load Testing.

TT's closing thought. Even if the kitchen stays the same and only the menu item names change, a floor manager who doesn't adapt will serve an empty plate instead of saying "We don't have that dish." That's why a PR fixing a single name can patch a surprisingly big hole!

References

All access dates are 2026-10-17.

• production-stack vllm-stack-0.1.13: src/vllm_router/, helm/, CONTRIBUTING.md, pyproject.toml
• vLLM v0.31.0: vllm/v1/metrics/loggers.py, kv_connector/v1/base.py, kv_connector/v1/lmcache_connector.py, kv_events.py, engine/arg_utils.py, docs/design/metrics.md, docs/usage/metrics.md
• LMCache v0.5.6: lmcache/integration/vllm/vllm_v1_adapter.py, lmcache/v1/token_database.py, lmcache/v1/storage_backend/, lmcache/v1/multiprocess/, CONTRIBUTING.md, AGENTS.md
• llm-d v0.10.0: COMPONENTS.md, docs/architecture/, docs/architecture/advanced/disaggregation/README.md, guides/env.sh, CONTRIBUTING.md
• llm-d-router v0.11.0: pkg/epp/, pkg/kvevents/, pkg/sidecar/
• Same series: LLM KV Cache Anatomy Part 2: KV Cache inside Serving Engines, Part 3: LMCache, Part 4: NVIDIA Dynamo

Cited code: vLLM, production-stack, and llm-d-router are all licensed under Apache License 2.0.