AI Storage and KV Cache Offload: From Weight Loading Paths to LMCache Practical Measurements

Status: Draft (Updated 2026-10-07), with some empirical measurements reflected. Based on official documentation, supplemented by Section 7's KV offload measurements (using a single 8GB-class GPU) and paper cross-checks in the "External Evidence" section. Storage performance (VAST, 3FS, GDS) was not directly measured, so only vendor data and paper conditions were verified. Read the "Verified Versions and Dates" and "Unverified Items" at the end of the post first. Writing rules: Empirical data comes first, and unmeasured items are marked as "Unmeasured".

Table of Contents

• 1. Why Loading is the Bottleneck
• 2. Hierarchy of Loading Paths
• 3. Model File Formats and Loading Optimization
• 4. High-Performance AI Storage: Focusing on VAST Data
• 5. Storage in Kubernetes
• 6. KV Cache Offload
• 7. Direct Empirical Measurements and Common Pitfalls
  • Advanced Measurements: Capacity Curves, Length-Based Gains, Disk Tiers, Chunk Sizes
  • External Evidence: Verified via Papers and Official Documentation
• 8. Hands-On
• 9. Potential Interview Questions
• 10. Common Misconceptions
• References (Actually Opened and Checked)
• Verified Versions and Dates (2026-10-05)

In large model serving, two issues frequently rival GPU computation in causing problems: the time required to move hundreds of gigabytes of weights from storage to GPU memory, and where to place the KV (Key-Value) cache when it does not fit entirely in GPU memory. This post groups these two problems into the storage tier (object storage, file systems, local NVMe, page cache, GPU memory). Because the author's measurements are on a small scale, their limitations are noted alongside.

What You Can Explain in an Interview After Reading This Post

1. Why loading becomes a bottleneck when hundreds of pods spin up simultaneously, and whether loader access patterns, file system layout, or cache tiers should be fixed first.
2. The problems solved by AI storage like VAST Data, what GPUDirect Storage (GDS) actually is, and which parts of vendor figures should be separated as "claims".
3. How to formulate the conditions where KV cache offload is beneficial as an equation, and how to plug in the author's measurements to calculate the break-even hit rate.

1. Why Loading is the Bottleneck

Weight size is the number of parameters multiplied by data type bytes. If 70 billion (70B) parameters are stored in bf16 (2 bytes), it is 140GB (calculated). Since every pod reads the same file, if 100 pods spin up simultaneously, the total amount to read becomes 14TB. Assuming an aggregate storage bandwidth of 10GB/s, reading alone takes about 1,400 seconds (23 minutes) (both are illustrative examples, not actual measurements).

In autoscaling, this time directly translates to latency. A pod must be scheduled, receive its image, load weights, complete compilation and warm-up before passing the readiness probe. If loading takes 10 minutes, newly scaled-up pods cannot accept requests for 10 minutes after a load spike, so loading time dictates choices across pre-scaling, headroom capacity, and loading path improvements.

NVIDIA Dynamo documentation provides a measured case. When a 554GiB model was loaded onto two 8xH100 nodes (tensor parallel 16) from Azure Managed Lustre (4 OSTs, aggregate ~8GB/s), the default loader achieved an effective throughput of about 124MB/s and took about 76 minutes to load. Changing the loader and striping files across all OSTs (Object Storage Targets) reduced this to about 5.3 minutes. This is a vendor measurement for a single environment. The key point is that even if storage has high bandwidth, a poor read access pattern by the client prevents utilizing it.

2. Hierarchy of Loading Paths

 객체 저장소 (S3 호환)  ─┐
 병렬/분산 파일 시스템   ─┼─▶ 노드 로컬 NVMe ─▶ 페이지 캐시 ─▶ CPU 버퍼 ─▶ GPU HBM
 (Lustre, VAST, WEKA 등) ─┘   (hostPath/로컬PV)   (DRAM, mmap)   (핀드 메모리)  (샤드별 적재)
   용량 큼, 공유, 느림  ◀──────────────────────────────────────▶  작고 빠름, 노드/GPU 전용

3. Model File Formats and Loading Optimization

safetensors format. The first 8 bytes (little-endian unsigned 64-bit) represent the header size N, the next N bytes are a JSON header containing tensor names, data types, shapes, and data offsets, and the remainder contains the raw tensor bytes. Unlike pickle, it does not execute code, making it secure; it allows lazy loading of only specific parts since the position of a tensor can be known just by reading the header. In tensor parallelism, get_slice allows reading only its own assigned slice. However, as the project README also notes, while cached files can be used zero-copy on the CPU, a copy to the GPU is strictly required once. Being "safe" refers to code execution and does not verify whether tensor values (such as NaNs) are correct.

Limitations of the default loader. According to Dynamo documentation, the vLLM default loader (--load-format auto) mmaps each safetensors shard and copies it via a single stream. Over remote storage, small requests go out with a shallow queue depth and get bottlenecked by latency. In Lustre measurements from the same documentation, single-target reads were about 228MB/s at 128KiB blocks, about 854MB/s at 1MiB, and about 1.6GB/s at 16MiB.

Streaming loaders. Run:ai Model Streamer in vLLM documentation is enabled via pip install vllm[runai] then --load-format runai_streamer . You adjust --model-loader-extra-config with concurrency(number of read threads), memory_limit(CPU buffer size), and distributed(distributed streaming from object storage) to read local paths, S3, GCS, and Azure Blob. If the model is pre-split into model-rank-{rank}-part-{part}.safetensors with runai_streamer_sharded , each worker reads only its own shard. This project was renamed to Model Streamer on 2026-09-13 and moved to dsx-ai-factory/model-streamer , but the PyPI package name remains unchanged.

Additionally, vLLM documentation mentions fastsafetensorsusing GDS, CoreWeave Tensorizer, and vllm preloadwhich remaps weights remaining in the GPU upon restart via CUDA IPC. Dynamo's ModelExpress receives weights via RDMA (Remote Direct Memory Access) from the GPU of a replica already serving, and the README claims 11 seconds (48x) compared to 8 minutes 53 seconds for cold pool under 8xB200 nodes, vLLM 0.23.0, and tensor parallelism 8 conditions (vendor figures).

4. High-Performance AI Storage: Focusing on VAST Data

According to the VAST whitepaper, the architecture is DASE (Disaggregated Shared-Everything). Stateless CNodes (protocol and logic servers running as containers) mount all SSDs via NVMe-oF (NVMe over Fabrics), and data is placed in SCM (write buffer) and QLC flash on DNodes/DBoxes. Access protocols include NFS (including NFS over RDMA), S3, SMB, and block. The NVIDIA GDS documentation lists network file systems capable of using GDS as DDN-EXAScaler, VAST-NFS, and WekaFS, and also includes Lustre as a supported target.

GDS is a path that sends data from storage directly to GPU memory via DMA (Direct Memory Access) without CPU bounce buffers. If conditions are not met (nvidia-fs missing driver, file system not supported, O_DIRECT unavailable), it falls back to a compatibility path passing through CPU memory. Therefore, "GDS support" does not mean it is enabled, but rather that it can be enabled, so the actual path must be verified.

Figures to Separate as Vendor Claims

• VAST blog (Dynamo integration, CoreWeave collaboration): Claims to maintain over 90% of line rate (181Gbps) during KV reads over a 200Gbps link using NFS over TCP (VAST NFS multipath driver), and achieves a TTFT (Time To First Token) drop from 62 seconds of prefill calculation to 3 seconds of KV load (20x) with Llama 3.1 405B (AWQ Int4) and a 127,188-token prompt. Calculated at 504KiB per token, this is about 65.6GB (61.1GiB; 61GB in the original text refers to GiB) and at 181Gbps (~22.6GB/s) it takes about 2.9 seconds, which matches the 3-second claim (author's recalculation, corrected to avoid mixing GB and GiB units).
• VAST-Lablup blog (2026-06-24): Reports an average TTFT reduction from 22,104ms to 10,573ms (2.09x) under 8xH100, Mistral Medium 3.5 128B, 140K token context, 100Gbps network, and GDS conditions, with a peak read throughput of 10,693MB/s and an average read latency of 8.265ms.
• The same article acknowledges that under regular NFS shares, the cost of fetching offloaded blocks can exceed GPU recomputation. This implies that gains hold true only when storage is sufficiently fast.
• VAST's KV storage library, the VUA store, was archived on 2026-07-28, and its connector functionality was integrated into LMCache's GDS backend, according to the README.

Alternative Comparison (Brief). Lustre is an open-source parallel file system where striping (stripe_count) dictates performance. DeepSeek's 3FS is a distributed file system (MIT license) based on NVMe and RDMA, storing metadata in FoundationDB; its README claims ~6.6TiB/s reads across 180 nodes (tested under a read stress test with 180 storage nodes and 500+ client nodes with background training traffic, noted in the original README as about 81% of the storage node NIC total). Besides the list in the GDS documentation, official materials for WEKA and DDN could not be accessed and verified.

5. Storage in Kubernetes

• CSI and ReadWriteMany (RWX). If multiple pods are to share the model cache, a storage class whose PersistentVolumeClaim (PVC) access mode is ReadWriteMany is required (where CSI stands for Container Storage Interface). The Dynamo documentation recommends a download job that prefills the RWX cache PVC and states, "Conduct scaling tests with multiple pods rather than a single pod."
• Freeloader pattern. A DaemonSet or Job prefetches the model onto node-local NVMe or a shared PVC, and the serving pods mount it read-only. This is a widely used design pattern rather than an official standard, and it incurs costs such as node pinning or node-specific copies.
• OCI image volume. volumes[].image.reference Mounts OCI images or artifacts as a read-only volume. According to the Kubernetes documentation, this was introduced in version 1.31, became stable (enabled by default) starting from 1.36, and requires container runtime support. It offers the advantage of reusing image layer caches and registries.
• KServe's LLMInferenceService supports HuggingFace, S3, and PVC as model locations.

6. KV Cache Offloading

        GPU HBM (G1)  ◀──────────────▶  CPU 핀드 메모리 (G2)
        활성 KV, 가장 빠름     DMA, 비동기       │  ▲
                                                 ▼  │ 선인출(prefetch)
                              로컬 NVMe/디스크 (G3) ◀──▶ 원격/객체 저장소 (G4)
   적중 시 위로 올림(promote), 넘치면 아래로 내림(LRU 등), GPU 와 직접 닿는 층은 CPU 층뿐

The tier names follow the Dynamo KVBM documentation (G1 GPU HBM, G2 pinned DRAM, G3 NVMe/SSD, G4 S3/MinIO). vLLM's OffloadingConnector has a similar architecture where only the CPU tier directly touches the GPU, and auxiliary tiers go through the CPU.

LMCache architecture. LMCache is a KV cache management layer, and the current documentation distinguishes between two modes. The original in-process mode operates inside vLLM LMCacheConnectorV1 and is marked as deprecated in the documentation. New deployments are recommended to use the multi-process (MP) mode, where an independent lmcache server runs per node (as a DaemonSet) and is shared by multiple vLLM pods via LMCacheMPConnector . The server maintains L1 (CPU DRAM or GDS NVMe slab) and L2 (file system, NIXL, S3, Redis, etc.), asynchronously pushing from L1 to L2 and promoting from L2 to L1 upon a miss. The documentation notes the advantages of eliminating GIL contention in the vLLM process and achieving fault isolation. The Kubernetes operator reconciles DaemonSet, Service, and ConfigMap with a single LMCacheEngine custom resource (lmcache.lmcache.ai/v1alpha1).

Prefix sharing among instances. There are four methods: pods on the same node share the L1 of an MP server, L2 is shared via a shared file system or S3 (using the adapter's "shared": true), vLLM's own fs tier is specified as the same root_dir on a shared PVC (hash seeds are fixed so identical file names are produced, and some hash algorithms require matching PYTHONHASHSEED ), or RDMA P2P sharing is used (which requires a coordinator and an RDMA network and cannot be used together with GDS L1). Since requests must go to the instance holding the cache to be beneficial, the scheduler in the KServe documentation indexes using BlockStored/BlockRemoved events sent by vLLM and routes based on a prefix hit score (weight 2.0) and a load score (1.0).

Cost versus benefit. The KV size per token is 2 × 층 수 × KV 헤드 수 × head_dim × 자료형 바이트. The author's experimental model Qwen2.5-1.5B (Hugging Face config: 28 layers, KV heads 2, head_dim 1536/12=128, fp16) is 28,672 bytes per token, which amounts to 139,460,608 bytes (0.1299 GiB) for 4,864 tokens. This matches the 0.1299 in the LMCache logs. A 70B-class Llama (80 layers, KV heads 8, head_dim 128, bf16) is 327,680 bytes per token, so 32,000 tokens is about 10.7 GB (calculated). The benefit of a single hit is T_재계산 − (S/B + 지연) , and the average gain with a hit probability p, p×이득 − (1−p)×저장비용 , must be positive. If the bandwidth B of the storage tier is low, S/B exceeds recomputation.

7. Direct Measurements and Common Pitfalls

KV offload measurement. Qwen2.5-1.5B-Instruct (fp16) was deployed on an 8GB-class laptop GPU using vLLM 0.29.0 and LMCache 0.5.5, with a GPU KV cache size of 21,520 tokens in both configurations. Ten documents (about 5,011 tokens per document, totaling about 50,000 tokens) were sent as individual requests in the order of first read, read again in the same order, and read in reverse order. LMCache was configured with a chunk size of 256, a CPU limit of 1.5GB, and the in-process mode (LMCacheConnectorV1).

• When read again in the same order, the time drops from 576 ms to 48.6 ms, which is about 11.9x faster. The gap is large because Least Recently Used (LRU) is the worst-case order for cyclic access, where even a slight shortage of space results in zero hits.
• The first read incurred about a 4% penalty (from 575.5 ms to 597.8 ms) due to storage overhead. Compared with external reports, this value is from a small configuration. A joint blog post by VAST and Lablup notes that for a 140K token context, the first-turn TTFT increased from about 22.3 seconds to about 26.3 seconds (an increase of about 18%, presumably caused by the LMCache implementation).
• From the logs, loading 4,864 tokens (0.13 GB) per request from the CPU took 8 to 9 ms, whereas recomputing the same amount took about 560 ms. The 8 to 9 ms is the timestamp difference in the log lines, not a separate timer. Converted, this is about 15 GB/s. When measuring the PCIe link of this GPU separately (Gen 4, x8 width, theoretical max ~15.75 GB/s), transferring 1 GiB of pinned memory from host to GPU was 13.4 GB/s, and from GPU to host was 13.2 GB/s (identical values across 3 repetitions, about 85% of theoretical), while regular (unpinned) memory was 11.3 GB/s. Since the log estimate of 8 to 9 ms is faster than the measured peak link throughput (13.4 GB/s), the log timestamp difference is a coarse estimate, and actual loading for 0.13 GB based on a max of 13.4 GB/s would take over 10 ms (calculated). In the advanced measurements above, the fact that loading for the length-specific value (6,414 tokens, KV ~0.18 GB) is about 9 ms slower than a GPU hit is on the same order of magnitude as the calculated transfer time at this bandwidth (~14 ms). They do not match precisely because parts of it overlap with computation (estimated). The average bandwidth required for loading is about 0.23 GB/s, obtained by transferring 0.13 GB per request within 576 ms, meaning what determines profit and loss is not transfer bandwidth, but recomputation time and hit rate (calculated).
• Let's calculate the break-even hit rate from the figures above. The hit benefit is 576.2 − 48.6 = 527.6 ms, and the miss penalty is 597.8 − 575.5 = 22.3 ms, so p* = 22.3 / (22.3 + 527.6) ≈ 4% (this calculation uses the p50 values from this experiment and is not a general constant. Plugging the values from the VAST/Lablup data into the same formula yields about 21%, varying greatly depending on the configuration).

Limitations of the experiment. Concurrency is 1, the model is small at 1.5B, the documents are synthetic documents generated with random words, and there is a single GPU. Standard deviations were not calculated since each value was measured only once, and because the image tag is latest , replicating it might yield different versions. Since in-process mode is currently deprecated in the LMCache documentation, figures for MP mode were not remeasured. The expectation that the performance gap will widen with larger models is not demonstrated by this measurement.

Advanced Measurements: Capacity Curves, Length-based Gains, Disk Tiers, and Chunk Sizes

The above experiment was rerun under changed conditions in the same environment (a single 8GB-class GPU, vLLM 0.29.0, LMCache 0.5.5, Qwen2.5-1.5B-Instruct). This time, the GPU KV pool was fixed at 21,520 tokens (the same value even after restarting the server), and there were 10 documents (5,189 tokens per document, totaling 52,025 tokens). The KV size for this working set is calculated as 28,672 bytes per token (28 layers × 2 KV heads × 128 head_dim × 2 for keys and values × 2 bytes), which is approximately 1.49 GB.

CPU cache capacity curve. Median TTFT when read again in the same order (LRU worst-case).

• If the limit is smaller than the working set (~1.49 GB), the gain is zero. At 0.5 GB and 1.0 GB, there was not a single hit. This is because under cyclic access, even a slight shortage of capacity causes LRU to evict the oldest items first, so the items needed for the next turn have already disappeared. The gains do not increase gradually with capacity but appear like a cliff.
• For the same reason, at the limit where only a single document can be held (0.5 GB), performance was slightly slower than using vLLM alone (first read increased from 599 ms to 618 ms, about 3%, paying only the storage overhead).

Skewed access case (60 documents, Zipfian distribution s=1, 60 requests). If certain documents are read frequently, a small cache also helps.

Since medians are mostly around 30 ms (frequently read documents hit directly in the GPU), the average and p95 better illustrate the capacity effect.

Gain by length (CPU 2 GB, requesting the same document again after being evicted from the GPU).

Recomputation time increases almost proportionally to length, whereas loading time increases much more gradually, meaning longer contexts yield greater gains. Loading takes only 1 to 17 ms longer than a GPU hit of the same length (20 to 32 ms).

Disk tier. Using 0.2 GB of CPU and 10 GB of local disk resulted in a re-read time of 57 ms (p95 255 ms), which was slower than using CPU only (39 ms) but much faster than recomputation (600 ms). Because the files were just written, they may have benefited from the operating system page cache, so this should not be interpreted as a cold disk value (unseparated).

Chunk size (64, 256, 1,024). When measuring prefix reuse on a document with a common 2,000-token prefix followed by 1,000 tail tokens, all three chunk sizes showed an external hit of 2,048 tokens and a TTFT of 140–144ms. No impact from chunk size was visible in this test. However, since the prefix in this test also remained in the GPU prefix cache and matched the value using vLLM alone (138ms), it is insufficient as a test to isolate the chunk alignment effect (unseparated).

Limitations. Measurements were taken one by one while varying server states (chunk and capacity curves were not repeated), with a concurrency of 1, using synthetic documents and a 1.5B model. When re-running the first experiment in Section 7 above under the same conditions, the initial read was slightly different at 598 to 618ms and the re-read at 48.6 to 39ms, so single-digit millisecond differences should not be overinterpreted.

Common Pitfalls: Dirty Pages and Memory Limits. When placing large objects on a network file system, dirty page caches that have not yet been flushed to storage may be counted toward container memory, causing memory limits to be exceeded and leading to OOMKilled errors. We have encountered cases resolved by limiting concurrent uploads and increasing the memory limit. This is an example of how the page cache layer intertwines with memory limits.

Unexplored Experience. I have not operated VAST, WEKA, or GDS. Section 4 is compiled based on official materials and vendor documentation.

External Evidence: Verified via Papers and Official Materials

KV offloading and storage descriptions were cross-referenced with primary sources. External figures were not directly compared due to differing models, GPUs, and loads; only consistency in direction and order of magnitude was evaluated.

8. Try It Yourself

All are read-only or end up as temporary pods.

# (1) 모델 설정만으로 토큰당 KV 크기 계산: 로컬 실행, 환경 변경 없음
python3 - <<'PY'
import json, urllib.request
u = "https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/resolve/main/config.json"
c = json.load(urllib.request.urlopen(u))
hd = c.get("head_dim") or c["hidden_size"] // c["num_attention_heads"]
b = 2 * c["num_hidden_layers"] * c["num_key_value_heads"] * hd * 2   # fp16/bf16
print(b, "B/token;", b * 4864 / 2**30, "GiB per 4864 tokens")
PY

# (2) 블록 크기에 따른 읽기 처리량: 기존 큰 파일을 읽기만 합니다(쓰기 없음)
for bs in 128k 1m 16m; do
  fio --name=r$bs --filename=/mnt/models/큰파일.safetensors --rw=read \
      --bs=$bs --direct=1 --size=4g --readonly | grep -E "READ:"
done

(3) Reproducing Experiments. Run vLLM and LMCache in a single temporary pod using GPUs, and delete the pod when finished. If node memory is shared with other workloads, do not arbitrarily increase the pod's memory limit.

9. Potential Interview Questions

1. How would you design model loading when hundreds of pods spin up simultaneously? Calculate the total read volume (number of pods × size) first, have readers use a streaming loader to read large blocks concurrently, and reduce shared storage load on the storage side through file layout, node-local caches, and inter-replica transfers. Validate with real-scale scale-out tests rather than a single pod.
2. Why is shard-by-shard loading possible in tensor parallelism? Because safetensors headers contain per-tensor offsets allowing only the required byte ranges to be read, and if files are pre-split by rank, each worker reads only its own file, reducing the total read volume.
3. Will adopting VAST automatically speed up loading and KV offloading? No. GDS is used only when conditions are met; otherwise, it falls back to compatibility mode, and poor loader access patterns fail to utilize bandwidth. Vendor benchmarks must be verified for conditions (GPU, model, network) and re-measured in your own environment.
4. When does KV cache offloading become detrimental? When the hit rate is below the breakeven point, storage layer bandwidth is low making fetch time longer than recalculation, or prefixes are short and different every time. In the author's actual measurements, initial reads suffered a ~4% penalty, and the breakeven hit rate was ~4%.
5. How can multiple instances share prefixes? Share KV via a shared L1 (MP server per node), shared L2 (NFS/S3), or P2P RDMA, and place a prefix-aware router so requests go to the instance holding the cache. If hash seeds do not match, the same prefix becomes different keys.

10. Common Misconceptions

• Fast storage means fast loading: Access patterns and file layout come first. Documentation notes that most Dynamo performance improvements also came from the loader.
• GDS is fast simply by turning it on: Compatibility mode fallbacks, driver, and file system conditions apply.
• KV offloading is always a gain: It is a trade-off dependent on the hit rate.
• LMCache is a library inside vLLM: The current recommendation is a separate server (MP mode).
• Vendor multipliers apply in my environment too: They change if models, context lengths, or network bandwidth differ.

References (Actually Opened)

• https://docs.lmcache.ai/ , /developer_guide/architecture.html , /mp/index.html , /mp/deployment.html , /mp/operator.html , /mp/p2p.html , /mp/l2_storage/index.html , /legacy/index.html
• https://github.com/LMCache/LMCache
• https://docs.vllm.ai/en/latest/features/kv_offloading_usage/ , /features/preload/ , /features/disagg_prefill/ , /models/extensions/runai_model_streamer/ , /models/extensions/fastsafetensor/
• https://github.com/run-ai/runai-model-streamer (previous announcement in README)
• https://github.com/huggingface/safetensors , https://huggingface.co/docs/safetensors/index
• https://github.com/ai-dynamo/dynamo/blob/v1.5.0/docs/fern/pages/kubernetes/model-deployment/model-loading/fast-model-loading.mdx
• https://github.com/ai-dynamo/dynamo/tree/v1.5.0/lib/kvbm-engine , https://github.com/ai-dynamo/modelexpress
• https://www.vastdata.com/whitepaper , https://www.vastdata.com/blog/vastdata-lablup-kvcache-offload-benchmark , https://www.vastdata.com/blog/nvidia-dynamo-vast-scalable-optimized-inference , https://github.com/vast-data/VUA
• https://docs.nvidia.com/gpudirect-storage/overview-guide/index.html
• https://github.com/deepseek-ai/3FS
• https://kubernetes.io/docs/concepts/storage/volumes/ , https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview , /concepts/architecture/control-plane-llmisvc

Verified Versions and Dates (2026-10-05)

LMCache v0.5.5 (2026-09-12), vLLM v0.30.0 (2026-09-22, docs.vllm.ai/en/latest is development documentation), Dynamo v1.5.0, Kubernetes v1.37.1, Model Streamer previous announcement 2026-09-13.

Unverified Items. WEKA and DDN official documentation, Lustre GDS detailed conditions, image volume support versions by container runtime, current official names and pricing of VAST products, MP mode performance (not remeasured), and differences between vLLM stable and development documentation.