Tag: #vllm

← All posts

• vLLM Metrics — What to Chart and What to Alert On

  The series vLLM exposes answer questions GPU metrics cannot: how many requests are running versus waiting right now, how full the KV cache is, how long until the first token. This post reads the official vLLM documentati

  2026-08-12 · 7 min read #gpu#kubernetes#vllm#prometheus#observability

• Inside vLLM (5) — Prefix Caching, and Why System Prompt Design Is Performance

  A rundown of when vLLM prefix caching hits and when it misses, based on the official design document. Covers how block hashes chain to the block before them, why only full blocks get cached, how a single timestamp placed

  2026-08-12 · 7 min read #vllm#prefix-caching#kv-cache#prompt-engineering#llm

• Inside vLLM (3) — How Continuous Batching Keeps the GPU Busy

  Explains why static batching leaves the GPU idle, and how vLLM iteration-level scheduling removes that idle time. Covers the difference in character between prefill and decode, V1 unified scheduler and its token budget,

  2026-08-12 · 7 min read #vllm#continuous-batching#chunked-prefill#llm#inference

• Inside vLLM (7) — Deployment Tuning, Common Pitfalls, and OOM Triage

  A practical order for tuning a vLLM deployment. Covers what gpumemoryutilization actually sets, when to use tensor parallelism versus pipeline parallelism, how to choose quantization and a KV cache data type, and a diagn

  2026-08-12 · 10 min read #vllm#gpu#quantization#tensor-parallel#llm

• Inside vLLM (2) — Why PagedAttention Splits the KV Cache Into Pages

  Explains why PagedAttention splits the KV cache into fixed-size blocks, verified against the original paper (arXiv:2309.06180) and the official vLLM design documentation. Covers the internal and external fragmentation th

  2026-08-12 · 7 min read #vllm#paged-attention#kv-cache#llm#gpu

• Inside vLLM (6) — Context Window vs max_model_len vs max_tokens, Fully Explained

  If the difference between context window and maxtokens has been confusing, this single post sorts it out. Covers what each of the following actually limits, with a comparison table and real error messages: the context wi

  2026-08-12 · 12 min read #vllm#context-window#max-model-len#max-tokens#llm

• Inside vLLM (4) — The Scheduler and Preemption, Where Throughput Collapses ♪ Listenable

  Explains what the vLLM scheduler decides on every step, how the waiting queue and running list move, and how preemption, triggered when the KV cache runs short, wrecks latency and throughput, verified against the officia

  2026-08-12 · 7 min read #vllm#scheduler#preemption#kv-cache#llm

• Inside vLLM (1) — The Full Path From One Request to One Token

  Follows the full path a single request takes through vLLM, from arrival to the first token out. Covers what the API server, scheduler, KV cache manager, worker, and sampler each do, and how the process layout splits afte

  2026-08-12 · 6 min read #vllm#llm#inference#gpu#ai-platform

• Making vLLM Fast — Configuration, Internals, and Where to Actually Touch the Code ♪ Listenable

  A step-by-step walk through improving vLLM performance, starting from everything you can fix without touching code. Covers batching-related arguments, prefix caching, chunked prefill, and quantization choice first, then

  2026-08-02 · 21 min read #vllm#llm-inference#paged-attention#benchmark#scheduler

• PD Disaggregation Doesn't Increase Throughput — What Prefill/Decode Disaggregation Actually Buys You

  PD disaggregation — splitting prefill and decode across separate GPUs — is a design that landed in vLLM, SGLang, and TensorRT-LLM alike in 2026, but wherever you look, all you find are numbers like "2x to 7x." Yet the of

  2026-07-16 · 16 min read #llm#ai#inference#kv-cache#vllm

• LLM Inference Serving 2026 — Comparing vLLM, SGLang, and TensorRT-LLM ♪ Listenable

  A clear overview of LLM inference serving in 2026. From core principles such as the difference in nature between prefill and decode, continuous batching, and paged KV cache, to a strengths-and-weaknesses comparison of vL

  2026-06-26 · 15 min read #llm-serving#vllm#sglang#tensorrt-llm#inference

• Serving Multimodal LLMs — The New Challenges Image Input Creates

  From how multimodal LLM serving differs from text-only serving, to the added vision-encoder stage, variable visual token counts, prefill cost spikes, the difficulty of multimodal KV cache and batching, latency decomposit

  2026-06-26 · 14 min read #mlops#multimodal#llm-serving#vllm#kv-cache

• The KV Cache and PagedAttention — Everything About Inference Memory

  A deep dive into the KV cache, the single biggest consumer of memory in LLM inference. We cover what the KV cache is and why it eats memory, the memory arithmetic and fragmentation problem, PagedAttention block managemen

  2026-06-26 · 12 min read #kv-cache#paged-attention#inference#gpu-memory#quantization

• Local LLM Inference Optimization — From Quantization to Breaking the VRAM Ceiling ♪ Listenable

  Privacy concerns, cost pressure, and big-tech fatigue are driving a local LLM revival. We map the entire landscape of local inference optimization: VRAM-first hardware thinking, GGUF and AWQ quantization, llama.cpp vs vL

  2026-06-12 · 16 min read #llm#inference#quantization#llama-cpp#vllm

• Top LLM Papers 2024-2026 - Llama, DeepSeek, Qwen, Mistral, Phi, RLHF, DPO, CoT, RAG, FlashAttention, vLLM Reading List

  A curated reading list of 30+ must-read LLM papers for engineers building with LLMs in 2024-2026. Covers foundation models (Llama 3/4, DeepSeek-V3/R1, Qwen3, Mistral, Phi-4, Gemma 3), training innovations (MoE, MLA, GQA)

  2026-05-16 · 19 min read #llm#papers#llama#deepseek#qwen

• LLM Serving & Local Inference in 2026 — vLLM / llama.cpp / MLX / Ollama / LM Studio / SGLang / TGI Deep Dive ♪ Listenable

  A map of the 2026 LLM serving and inference landscape. Datacenter camp (vLLM, SGLang, TGI, Triton, TensorRT-LLM), local camp (llama.cpp, MLX, llamafile, Ollama, LM Studio, GPT4All), emerging camp (KTransformers, MLC LLM,

  2026-05-16 · 25 min read #llm#model-serving#inference#vllm#llama-cpp

• AI Inference Engines 2026 - vLLM · SGLang · llama.cpp · TGI · TensorRT-LLM · MLX · mistral.rs · DeepSpeed-MII · Aphrodite Deep Dive

  In 2026, LLM engineering is no longer about which model — it is about which inference engine. We dissect vLLM V1, SGLang 0.4, TensorRT-LLM, TGI 3.x, llama.cpp, MLX-LM, mistral.rs, DeepSpeed-MII, Aphrodite, CTranslate2, E

  2026-05-16 · 20 min read #llm-inference#vllm#sglang#llama-cpp#tgi

• MLOps Complete Guide — Model Serving, Feature Store, Drift, A/B Testing, GPU Economics (Season 2 Ep 7, 2025)

  Training a model and running it in production are completely different games. Serving (TorchServe, Triton, vLLM, TGI), Feature Stores (Feast, Tecton), training infra (Ray, Determined), experiment tracking (MLflow, W&B),

  2026-04-15 · 12 min read #mlops#model-serving#feature-store#drift-detection#ab-testing

• Complete LLM Inference Optimization Guide 2025: vLLM, TensorRT-LLM, KV Cache, Speculative Decoding ♪ Listenable

  Everything about LLM inference optimization! vLLM (PagedAttention), TensorRT-LLM (FP8/INT4), KV Cache management, Speculative Decoding, Continuous Batching, FlashAttention, Quantization (GPTQ/AWQ/GGUF), Model Serving (Tr

  2026-04-14 · 25 min read #llm-inference#vllm#tensorrt-llm#kv-cache#speculative-decoding

• Complete LLM Serving Optimization Guide: KV Cache, PagedAttention, and Quantization ♪ Listenable

  A complete engineering deep dive into LLM serving optimization. From the KV cache memory crisis to the virtual-memory revolution of PagedAttention, continuous batching, speculative decoding, quantization techniques, and

  2026-03-18 · 30 min read #model-serving#kv-cache#paged-attention#vllm#quantization