Tag: #inference

← All posts

• Cerebras Ultrafast and the Bottleneck in an Agent Loop — The Time 750 Tokens a Second Cannot Remove

  Cerebras and OpenAI announced an inference tier delivering up to 750 output tokens per second. The mechanism the announcement names is not compute but data movement, achieved by keeping weights in SRAM on the wafer. This

  2026-08-14 · 7 min read #llm#inference#hardware#latency#performance

• The Hybrid Attention in Qwen3.8-27B — Only 16 of 64 Layers Grow a KV Cache ♪ Listenable

  The reason a 27B model can carry a 262K-token context on a laptop is not its parameter count but its layer layout. Qwen3.8-27B places 48 of its 64 layers as linear attention (Gated DeltaNet) and only 16 as ordinary atten

  2026-08-14 · 8 min read #llm#inference#quantization#local-llm#attention

• Open Source Worth Watching Right Now (1) AI Agents and LLM Tooling

  The LLM application stack has split into layers: inference servers, orchestration, gateways, agents, and RAG. This post introduces 12 open source projects that are actually used at each layer, grouped by role rather than

  2026-08-12 · 6 min read #open-source#llm#ai-agent#ai-platform#rag

• Choosing Open Text Generation Models by Size Class

  When picking an open text generation model, size is not a quality tier — it is a deployment constraint. This post splits the field into on-device small, single-GPU mid, and server-class large, listing the parameters, con

  2026-08-12 · 7 min read #ai#llm#huggingface#open-source-llm#text-generation

• Inside vLLM (3) — How Continuous Batching Keeps the GPU Busy

  Explains why static batching leaves the GPU idle, and how vLLM iteration-level scheduling removes that idle time. Covers the difference in character between prefill and decode, V1 unified scheduler and its token budget,

  2026-08-12 · 7 min read #vllm#continuous-batching#chunked-prefill#llm#inference

• Inside vLLM (1) — The Full Path From One Request to One Token

  Follows the full path a single request takes through vLLM, from arrival to the first token out. Covers what the API server, scheduler, KV cache manager, worker, and sampler each do, and how the process layout splits afte

  2026-08-12 · 6 min read #vllm#llm#inference#gpu#ai-platform

• Reasoning Effort Is Not a Model Choice but a Per-Request Deployment Parameter

  The DeepSeek V4 Flash 0731 results page published by ARC Prize carries not one score but three, one per reasoning effort level. This post computes what can actually be read out of those three numbers: that the same step

  2026-08-09 · 8 min read #llm#benchmark#arc-agi#inference#cost

• LLM Inference VRAM Math — the KV Cache Blows Up Before the Weights Do

  How to answer "will this model fit on our GPU" with a few multiplications. Weight memory is just parameter count times bytes per parameter, but the thing that actually blocks a deployment is the KV cache, which grows wit

  2026-07-26 · 12 min read #llm#inference#vram#kv-cache#quantization

• PD Disaggregation Doesn't Increase Throughput — What Prefill/Decode Disaggregation Actually Buys You

  PD disaggregation — splitting prefill and decode across separate GPUs — is a design that landed in vLLM, SGLang, and TensorRT-LLM alike in 2026, but wherever you look, all you find are numbers like "2x to 7x." Yet the of

  2026-07-16 · 16 min read #llm#ai#inference#kv-cache#vllm

• More Thinking Is Not More Accuracy: Test-Time Compute and the Overthinking Cliff

  In April 2026, Shu Zhou and five co-authors question the reflex to keep adding reasoning tokens in "When More Thinking Hurts". As the compute budget grows, the authors report that the marginal utility of extra reasoning

  2026-07-11 · 6 min read #ai#llm#reasoning#inference#test-time-compute

• Quantizing the KV Cache to 4 Bits — SAW-INT4 and System-Aware INT4

  In long-context LLM serving, the real memory bottleneck is not the weights but the KV cache. Quantize it naively to INT4 and accuracy collapses; the April 2026 preprint SAW-INT4 pairs token-wise INT4 quantization with a

  2026-07-11 · 6 min read #ai#llm#quantization#inference#kv-cache

• Optimizing Long-Context Inference with Hybrid SWA — What Xiaomi's MiMo v2.5 Actually Does

  Hybrid SWA interleaves sliding-window attention with full attention across layers to cut both KV-cache memory and compute for long-context inference — now a mainstream design. Grounded in Xiaomi's published MiMo v2.5 inf

  2026-07-11 · 7 min read #ai#llm#inference#attention#optimization

• Why the Mac mini Became an On-Device AI Machine — What Apple's Silicon Exec Said, and What He Left Out

  Apple Silicon senior product manager Doug Brooks talked to The Deep View about demand for the Mac mini and Mac Studio and where on-device AI is heading. Why developers and small teams reach for this little desktop as a l

  2026-07-11 · 6 min read #apple-silicon#on-device-ai#local-llm#mac-mini#inference

• LLM Caching, Explained — Why Prompt Caching and Prefix Caches Save You Money

  Why does a matching prompt prefix cut costs to a tenth? The answer lives in the transformer's KV cache. Because attention is causal, the Key/Value vectors of earlier tokens never change no matter what comes after — so th

  2026-07-08 · 7 min read #ai#llm#caching#inference#performance

• The State of LLM Quantization — From GPTQ and AWQ to FP8, MXFP4, and KV-Cache Quantization ♪ Listenable

  Quantization represents a model's numbers in fewer bits to cut memory and cost. Why quality survives fewer bits (outliers and scaling), how GPTQ and AWQ differ in approach, llama.cpp GGUF k-quants, QLoRA's NF4, the 2026

  2026-07-08 · 7 min read #ai#llm#quantization#inference#optimization

• LLM Inference Serving 2026 — Comparing vLLM, SGLang, and TensorRT-LLM ♪ Listenable

  A clear overview of LLM inference serving in 2026. From core principles such as the difference in nature between prefill and decode, continuous batching, and paged KV cache, to a strengths-and-weaknesses comparison of vL

  2026-06-26 · 15 min read #llm-serving#vllm#sglang#tensorrt-llm#inference

• Making Inference Fast — Speculative Decoding and Throughput Optimization

  From the fundamental reason LLM decode is slow, to how speculative decoding boosts speed, variants such as Medusa and EAGLE, chunked prefill and prefill/decode disaggregation, the latency versus throughput trade-off, and

  2026-06-26 · 13 min read #speculative-decoding#throughput#inference#mlops#latency

• Serving Multimodal LLMs — The New Challenges Image Input Creates

  From how multimodal LLM serving differs from text-only serving, to the added vision-encoder stage, variable visual token counts, prefill cost spikes, the difficulty of multimodal KV cache and batching, latency decomposit

  2026-06-26 · 14 min read #mlops#multimodal#llm-serving#vllm#kv-cache

• The KV Cache and PagedAttention — Everything About Inference Memory

  A deep dive into the KV cache, the single biggest consumer of memory in LLM inference. We cover what the KV cache is and why it eats memory, the memory arithmetic and fragmentation problem, PagedAttention block managemen

  2026-06-26 · 12 min read #kv-cache#paged-attention#inference#gpu-memory#quantization

• Custom ASIC vs GPU: An Investing Perspective on the Chip War ♪ Listenable

  An investing-focused analysis of the rise of cloud providers in-house chips (custom ASICs) and the shifting inference market. We weigh Nvidia moat and the challenges to it, map the value-chain beneficiaries, and lay out

  2026-06-18 · 22 min read #asic#gpu#semiconductor#investing#nvidia