Tag: #kv-cache

← All posts

• Inside vLLM (5) — Prefix Caching, and Why System Prompt Design Is Performance

  A rundown of when vLLM prefix caching hits and when it misses, based on the official design document. Covers how block hashes chain to the block before them, why only full blocks get cached, how a single timestamp placed

  2026-08-12 · 7 min read #vllm#prefix-caching#kv-cache#prompt-engineering#llm

• Inside vLLM (2) — Why PagedAttention Splits the KV Cache Into Pages

  Explains why PagedAttention splits the KV cache into fixed-size blocks, verified against the original paper (arXiv:2309.06180) and the official vLLM design documentation. Covers the internal and external fragmentation th

  2026-08-12 · 7 min read #vllm#paged-attention#kv-cache#llm#gpu

• Inside vLLM (4) — The Scheduler and Preemption, Where Throughput Collapses ♪ Listenable

  Explains what the vLLM scheduler decides on every step, how the waiting queue and running list move, and how preemption, triggered when the KV cache runs short, wrecks latency and throughput, verified against the officia

  2026-08-12 · 7 min read #vllm#scheduler#preemption#kv-cache#llm

• Attention Variants — From MHA to MLA, and How the KV Cache Shrinks

  Comparing MHA, MQA, GQA, and MLA using real config values. Working out with formulas and numbers how the grouped-query attention in Qwen3, Mixtral, and GLM-4.5 and the latent attention in DeepSeek-V3 and Kimi K2 reduce p

  2026-08-12 · 6 min read #ai-papers#model-internals#attention#gqa#mla

• LLM Inference VRAM Math — the KV Cache Blows Up Before the Weights Do

  How to answer "will this model fit on our GPU" with a few multiplications. Weight memory is just parameter count times bytes per parameter, but the thing that actually blocks a deployment is the KV cache, which grows wit

  2026-07-26 · 12 min read #llm#inference#vram#kv-cache#quantization

• How Much VRAM to Run an LLM Locally — Compute It From Formulas, Not Tables ♪ Listenable

  The answer to "how many GB does an 8B model need?" is not a table but two formulas. Weights are parameter count times bpw divided by 8; the KV cache is 2 times layers times KV heads times headdim times bytes times tokens

  2026-07-17 · 24 min read #llm#quantization#kv-cache#local-llm#gpu

• PD Disaggregation Doesn't Increase Throughput — What Prefill/Decode Disaggregation Actually Buys You

  PD disaggregation — splitting prefill and decode across separate GPUs — is a design that landed in vLLM, SGLang, and TensorRT-LLM alike in 2026, but wherever you look, all you find are numbers like "2x to 7x." Yet the of

  2026-07-16 · 16 min read #llm#ai#inference#kv-cache#vllm

• Quantizing the KV Cache to 4 Bits — SAW-INT4 and System-Aware INT4

  In long-context LLM serving, the real memory bottleneck is not the weights but the KV cache. Quantize it naively to INT4 and accuracy collapses; the April 2026 preprint SAW-INT4 pairs token-wise INT4 quantization with a

  2026-07-11 · 6 min read #ai#llm#quantization#inference#kv-cache

• Serving Multimodal LLMs — The New Challenges Image Input Creates

  From how multimodal LLM serving differs from text-only serving, to the added vision-encoder stage, variable visual token counts, prefill cost spikes, the difficulty of multimodal KV cache and batching, latency decomposit

  2026-06-26 · 14 min read #mlops#multimodal#llm-serving#vllm#kv-cache

• Dissecting the Transformer — From Attention to KV Cache

  A from-scratch breakdown of the Transformer: self-attention, multi-head, positional encoding, the FFN, and residual connections with normalization. It connects tensor shapes and parameter counts, causal masking, encoder/

  2026-06-26 · 16 min read #llm#transformer#attention#positional-encoding#kv-cache

• The Evolution of Attention — MQA, GQA, FlashAttention, and Long Context

  We analyze the memory and compute cost of standard attention, then explain how MQA and GQA shrink the KV cache and how FlashAttention optimizes IO. We compare sliding-window and long-context techniques and trace how all

  2026-06-26 · 18 min read #llm#attention#flashattention#gqa#mqa

• The KV Cache and PagedAttention — Everything About Inference Memory

  A deep dive into the KV cache, the single biggest consumer of memory in LLM inference. We cover what the KV cache is and why it eats memory, the memory arithmetic and fragmentation problem, PagedAttention block managemen

  2026-06-26 · 12 min read #kv-cache#paged-attention#inference#gpu-memory#quantization

• Complete LLM Inference Optimization Guide 2025: vLLM, TensorRT-LLM, KV Cache, Speculative Decoding ♪ Listenable

  Everything about LLM inference optimization! vLLM (PagedAttention), TensorRT-LLM (FP8/INT4), KV Cache management, Speculative Decoding, Continuous Batching, FlashAttention, Quantization (GPTQ/AWQ/GGUF), Model Serving (Tr

  2026-04-14 · 25 min read #llm-inference#vllm#tensorrt-llm#kv-cache#speculative-decoding

• Complete LLM Serving Optimization Guide: KV Cache, PagedAttention, and Quantization ♪ Listenable

  A complete engineering deep dive into LLM serving optimization. From the KV cache memory crisis to the virtual-memory revolution of PagedAttention, continuous batching, speculative decoding, quantization techniques, and

  2026-03-18 · 30 min read #model-serving#kv-cache#paged-attention#vllm#quantization

• NPU Deep Dive: How Transformer Architecture Runs Directly on Silicon ♪ Listenable

  A complete technical breakdown of how NPUs differ from CPUs and GPUs, how every transformer operation maps to hardware, and why LLM inference is memory-bound — not compute-bound. From Apple ANE to Qualcomm Hexagon, Groq

  2026-03-18 · 22 min read #npu#transformer#ai-hardware#quantization#kv-cache

• LLM Inference Optimization Complete Guide: KV Cache, Speculative Decoding, Continuous Batching ♪ Listenable

  A complete guide to pushing LLM inference to its limits. Covers KV Cache, Speculative Decoding, Continuous Batching, PagedAttention, FlashInfer, multi-GPU inference, and DeepSeek MLA in depth.

  2026-03-17 · 25 min read #llm#inference#optimization#kv-cache#speculative-decoding

• The Complete Guide to LLM Inference Optimization: vLLM, TensorRT-LLM, Speculative Decoding ♪ Listenable

  A comparative analysis of key technologies for maximizing LLM inference performance — vLLM, TensorRT-LLM, Speculative Decoding, and KV Cache optimization — with practical code examples and benchmarks.

  2026-03-14 · 19 min read #llm#inference-optimization#vllm#tensorrt-llm#speculative-decoding

• KV Cache Optimization Deep Dive: GQA, MLA, and MHA Attention Mechanisms with Memory Efficiency Strategies

  From Transformer Self-Attention KV Cache fundamentals through MHA, MQA, GQA (Llama 2/3), and MLA (DeepSeek-V2/V3) mechanism memory analysis and comparison, KV Cache compression techniques (quantization, eviction policies

  2026-03-11 · 17 min read #ai-papers#kv-cache#attention-mechanism#gqa#mla

• Complete Guide to LLM Long-Context Performance and KV Cache Optimization: From MQA to Ring Attention

  A comprehensive practical guide covering KV Cache fundamentals for LLM long-context processing, memory consumption analysis, optimization techniques including MQA, GQA, PagedAttention, sliding window attention, and Ring

  2026-03-07 · 17 min read #llm#kv-cache#long-context#multi-query-attention#grouped-query-attention