Tag: #llm-inference
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 7 posts
7 posts are tagged #llm-inference; the most recent was published on 2026-08-12.
Often tagged together: #vllm 5 #gpu 2 #kv-cache 2 #llama-cpp 2 #model-serving 2
Most read in the last 90 days:
- AI Inference Engines 2026 - vLLM · SGLang · llama.cpp · TGI · TensorRT-LLM · MLX · mistral.rs · DeepSpeed-MII · Aphrodite Deep Dive
- Complete LLM Inference Optimization Guide 2025: vLLM, TensorRT-LLM, KV Cache, Speculative Decoding
- llama.cpp Comes to the Browser — The Ceiling a WebGPU Backend Measured Across 16 Devices
vLLM Metrics — What to Chart and What to Alert On
The series vLLM exposes answer questions GPU metrics cannot: how many requests are running versus waiting right now, how full the KV cache is, how long until the first token. This post reads the official vLLM documentati
2026-08-12 · 7 min read #gpu#kubernetes#vllm#prometheus#observabilityMaking vLLM Fast — Configuration, Internals, and Where to Actually Touch the Code ♪ Listenable
A step-by-step walk through improving vLLM performance, starting from everything you can fix without touching code. Covers batching-related arguments, prefix caching, chunked prefill, and quantization choice first, then
2026-08-02 · 21 min read #vllm#llm-inference#paged-attention#benchmark#schedulerllama.cpp Comes to the Browser — The Ceiling a WebGPU Backend Measured Across 16 Devices
LlamaWeb, the WebGPU backend for llama.cpp released by a UC Santa Cruz team in May 2026, left behind the widest dataset to date measuring browser LLM inference — 16 devices across 8 vendors. The results cut both ways. It
2026-07-16 · 16 min read #webgpu#llm-inference#llama-cpp#on-device-ai#browserAI Inference Engines 2026 - vLLM · SGLang · llama.cpp · TGI · TensorRT-LLM · MLX · mistral.rs · DeepSpeed-MII · Aphrodite Deep Dive
In 2026, LLM engineering is no longer about which model — it is about which inference engine. We dissect vLLM V1, SGLang 0.4, TensorRT-LLM, TGI 3.x, llama.cpp, MLX-LM, mistral.rs, DeepSpeed-MII, Aphrodite, CTranslate2, E
2026-05-16 · 20 min read #llm-inference#vllm#sglang#llama-cpp#tgiComplete LLM Inference Optimization Guide 2025: vLLM, TensorRT-LLM, KV Cache, Speculative Decoding ♪ Listenable
Everything about LLM inference optimization! vLLM (PagedAttention), TensorRT-LLM (FP8/INT4), KV Cache management, Speculative Decoding, Continuous Batching, FlashAttention, Quantization (GPTQ/AWQ/GGUF), Model Serving (Tr
2026-04-14 · 25 min read #llm-inference#vllm#tensorrt-llm#kv-cache#speculative-decodingNPU Deep Dive: How Transformer Architecture Runs Directly on Silicon ♪ Listenable
A complete technical breakdown of how NPUs differ from CPUs and GPUs, how every transformer operation maps to hardware, and why LLM inference is memory-bound — not compute-bound. From Apple ANE to Qualcomm Hexagon, Groq
2026-03-18 · 22 min read #npu#transformer#ai-hardware#quantization#kv-cacheThe New Throne Beyond vLLM: 5 Reasons SGLang Is Changing the LLM Inference Landscape ♪ Listenable
From SGLang's RadixAttention-based KV cache management and 29% throughput advantage through hyper-specialized design, to its 4,000-line Python zero-overhead scheduler, Prefill-Decode Disaggregation, and Compressed FSM-ba
2026-03-01 · 35 min read #sglang#llm-inference#vllm#radix-attention#model-serving