Operational Features: Sleep Mode, Startup Time, /metrics, and the Paths That --api-key Ignores

Status: Draft (2026-10-27), measured empirically. Measured on 2026-10-27 using a single 8GB-class laptop GPU (Ada generation), vLLM 0.31.0, and Qwen3-Reranker-0.6B. One container died due to the memory limit during sleep mode tests, and this incident has been included as part of the results. Details on metric name changes across versions can be found in Part 3 of "Open Source Contribution Log," and OpenTelemetry tracing is covered in Part 25 of "Preparing Toss ML/AI Infra." Added on 2026-10-27. Results from re-measuring authentication paths, startup times, and sleep modes with vLLM 0.32.0 on the same day have been added to the "0.32.0 Re-measurement" section. The 0.31.0 figures remain unchanged. Writing Rules: Empirical data is the baseline; unmeasured items are marked as "Unmeasured," and unverified items are marked as "Unverified."

Table of Contents

• 1. Sleep Mode: Clearing the GPU Without Stopping the Server
  • Direct Measurement
• 2. Startup Time: Compile Caches Cut It in Half
• 3. /metrics: What Is Exposed
• 4. --api-key Only Protects /v1
• 5. Key Takeaways for Operations
• 6. Unverified Items
• References

1. Sleep Mode: Clearing the GPU Without Stopping the Server

To share a single GPU between multiple models or training tasks, GPU memory must be released when not in use. Shutting down the server takes dozens of seconds to restart (Section 2). Sleep mode releases large amounts of memory while keeping processes, CUDA contexts, and compiled graphs intact.

Enabling sleep mode requires both server options and dev mode environment variables. Because sleep and wake operations are exposed via HTTP endpoints, they only appear in development mode.

VLLM_SERVER_DEV_MODE=1 vllm serve <모델> --enable-sleep-mode
curl -X POST 'localhost:8000/sleep?level=1'
curl -X POST 'localhost:8000/wake_up'
# 2단계: 가중치 자리만 깨우고, 가중치를 다시 읽고, KV 를 깨운다
curl -X POST 'localhost:8000/sleep?level=2'
curl -X POST 'localhost:8000/wake_up?tags=weights'
curl -X POST 'localhost:8000/collective_rpc' -H 'Content-Type: application/json' -d '{"method":"reload_weights"}'
curl -X POST 'localhost:8000/wake_up?tags=kv_cache'

--enable-sleep-mode enables vLLM's CuMem allocator, which allocates weights and KV caches to tagged virtual memory, releasing and reattaching only the physical memory. The vLLM blog (October 26, 2025, vLLM 0.11.0) reported that waking up Qwen3-0.6B from Phase 1 took 0.26 seconds and Phase 2 took 0.85 seconds on an A100, which is 18 to 20 times faster than a fresh start. In the same post, an A4000 took between 0.1 and 0.8 seconds to wake up (vendor announcement).

Direct Measurement

• Environment: Single 8GB-class laptop GPU (Ada generation), vLLM 0.31.0, Qwen3-Reranker-0.6B (1.12GiB weights), VLLM_SERVER_DEV_MODE=1 --enable-sleep-mode.
• Method: We first sent a request using greedy decoding to generate 100 tokens, recorded the output hash, and then repeated the cycle of sleep → wake → identical request four times. We recorded the HTTP call duration for each phase, nvidia-smi GPU memory, and the container cgroup's anonymous memory. Phase 2 was tested first, followed by Phase 1.

Here is how to interpret these results.

1. Phase 2 was virtually free for this model. Even when reading the 1.12GiB weights back from disk (page cache), it woke up within 0.4 seconds, and the first request after waking produced the same output in the same amount of time as before sleeping. This is about 80 times faster than a restart with the same settings (Section 2, 32 seconds) because the CUDA context and compiled graphs remain alive.
2. 606MiB remained allocated even during sleep. This is the CUDA context, CUDA graphs, and buffers not managed by the sleep allocator. The GPU memory available for other tasks is the total memory minus this amount.
3. Phase 1 exceeded the container memory limit. Phase 1 moves weights to CPU pinned memory, which increases container memory usage accordingly. Theoretically, 2.75GiB + 1.12GiB should have fit well within the 4.5GiB limit, but it exceeded it both times. It appears that additional data is moved besides the weights, or temporary duplicates are created during the copy process; the exact amount is unverified. While this OOM only affected a single pod and did not impact the node, a similar incident in production would crash the serving engine entirely. To use Phase 1, you must provision a container limit with an extra margin of at least twice the weight size and test thoroughly beforehand. 0.32.0 Re-measurement (Measured with 0.31.0 on 2026-10-27 → Re-measured with 0.32.0). The 0.32.0 release notes state that sleep mode was fixed to release workspaces and KV initialization states as well. We re-measured using the same model and the same 4.5GiB limit.

Phase 2 remained almost identical even when the setup changed, and the remaining 606MiB during sleep did not decrease either. The memory issue from Phase 1 also remained as it was. All three OOMs ended in a single pod and did not affect the node.

4. The documentation warns about the same thing. Phase 1 backs up weights to the CPU, so there must be sufficient CPU memory. If CPU memory is insufficient, it advises using Phase 2 and reading again with reload_weights . This actual measurement shows that "sufficient" is larger than the weight size.

2. Startup Time: Compilation cache halved it

Starting the server multiple times in the same pod, we recorded the time until /health reached 200.

vLLM leaves the torch.compile results under ~/.cache/vllm , and re-reads them if the model, settings, related environment variables, torch version, and GPU are the same. The log shows Directly load AOT compilation from path … . Since this cache disappears when a new container starts, the documentation recommends placing the cache folder on a volume or baking it into the image if it is a server that restarts frequently. Version 0.31.0 documentation additionally lists --kv-cache-memoryto skip memory measurement, --enforce-eagerto disable compilation entirely (making decoding slower instead), and optimization levels from -O0 to -O3 (default -O2) as levers for startup time.

Aside from startup time, even after the server outputs "ready", compilation happens again for newly encountered types of workloads. As seen in part 4, compiling the Triton kernel during the first sampling batch took several seconds, and a jit_monitor warning remained in the log. It is safer to issue the readiness check after warming up once with the actual request shape rather than /health .

Re-measurement on 0.32.0. It showed the same pattern as 0.31.0, taking 60 seconds for the first startup in a new container and 30 seconds for restarting with the same settings. In 0.32.0 logs, a "JIT kernel warmup" phase was newly visible, but the kernel compilation warning and delay in the first sampling batch (6.2s for chat model, 5.3s at concurrency 16 for re-ranking model) remained as they were.

3. /metrics: What comes out

/metrics is in Prometheus format. In the default server, there were 38 types of metrics starting with vllm: , while the speculative decoding server had 41 types, and the KV offloading server had 53 types. We grouped the ones worth placing on an operational dashboard first, along with actual values from the measured server.

Metric names have changed across versions. For instance, in the V0 engine, vllm:gpu_prefix_cache_hit_rate was removed in 0.10, and its place is taken by the two counters hits and queries mentioned above. Dashboards and alert rules must be checked against the names currently emitted by the server every time the version is upgraded. The method used this time is simple: curl -s localhost:8000/metrics | grep '^# HELP vllm:' was used to extract the list of names and compare it with the rule files.

4. --api-key only secures /v1

vLLM's security documentation clearly states that 「--api-key secures only paths starting with /v1, /v2, /inference, /cohere 」 and lists the paths that remain open without authentication. We verified if this is actually the case. We started the server with VLLM_API_KEY(a random value used and discarded only for this test) and called each path without a key.

0.32.0 remeasurement (measured 0.31.0 on 2026-10-27 → remeasured 0.32.0). When I ran the same test again in the official 0.32.0 container, not a single status code out of the 14 paths changed. /invocations still returned 200 without a key in 0.32.0.

/invocations It leads to one conclusion. If you expose this server as-is to the internet or another team's network, anyone can run the model even if you provided a key. documentation should --api-key not rely solely on , and it is recommended to place a reverse proxy or gateway in front that allows only permitted paths. According to the same document, the pooling model's /score, /rerank, /pooling, /classify also /v1 without authentication is not accepted.

Development mode (VLLM_SERVER_DEV_MODE=1) is even more dangerous. It opens up /collective_rpc which allows putting the engine to sleep (denial of service), flushing the cache, and calling arbitrary methods inside the engine. The documentation states that it should never be enabled in production. If you want to use the sleep mode from Section 1 in production, you must block its path with network policies so it is not reachable from the outside, and only allow the orchestrator to call it.

0.32.0 changed the default CORS from "allow all origins" to "no origins allowed". If your service was called directly from the browser, you must specify --allowed-origins when upgrading the version (based on the release notes, unmeasured).

5. Takeaways for Operations

• Test sleep mode starting from level 2. To use level 1, test it first by adding at least twice the weight size to the container memory limit. In our actual measurements, containers that lacked this headroom died entirely with OOM twice.
• Placing the compilation cache on a volume cuts restart times in half for the same configuration. Rebuilding the environment or changing options causes a cache mismatch, triggering a re-compilation.
• Readiness probes should be issued only after warming up with actual request patterns.
• /metrics Verify names for each version.
• Always place a route-selecting gateway in front of vLLM. --api-key only respects /v1 .

6. Unverified Items

• Wake-up times for tensor parallelism (multiple GPUs) and large models in sleep mode were not measured.
• Improvements in vLLM 0.32.0 (workspace/KV initialization state release, CUDA graph offloading) and changes to default CORS settings were not measured.
• /pause·/resume The conditions under which this returned 404 in this version (such as whether separate required options exist) remain unverified.

References

• vLLM Documentation, Sleep Mode, Optimization and Tuning (Faster Startup), Production Metrics, Security (0.32.0): https://docs.vllm.ai/en/latest/
• vLLM Blog, "Zero-Reload Model Switching with vLLM Sleep Mode", October 26, 2025: https://vllm.ai/blog/2025-10-26-sleep-mode
• vLLM 0.32.0 Release Notes (CORS, Sleep Mode, Histogram Buckets): https://github.com/vllm-project/vllm/releases/tag/v0.32.0
• Previous Posts: "Open Source Contribution Log" Part 3: Metric Names Removed by vLLM, "Toss ML/AI Infra Preparation" Part 25: Observing LLM Serving with OpenTelemetry