Tag: #latency
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 5 posts
5 posts are tagged #latency; the most recent was published on 2026-08-14.
Often tagged together: #inference 3 #llm 2 #performance 2 #speculative-decoding 2 #throughput 2
Most read in the last 90 days:
Cerebras Ultrafast and the Bottleneck in an Agent Loop — The Time 750 Tokens a Second Cannot Remove
Cerebras and OpenAI announced an inference tier delivering up to 750 output tokens per second. The mechanism the announcement names is not compute but data movement, achieved by keeping weights in SRAM on the wafer. This
2026-08-14 · 7 min read #llm#inference#hardware#latency#performanceThe Mean Tells You Nothing — How to Debug Latency Through Distributions
Farid Zakaria's post "The mean means nothing," published July 27, 2026 and reaching the top of Hacker News, covers a case where average latency got 9% worse after a caching layer rollout — from 112ms to 122ms. In the ver
2026-07-31 · 14 min read #observability#latency#performance#histogram#percentileMaking Inference Fast — Speculative Decoding and Throughput Optimization
From the fundamental reason LLM decode is slow, to how speculative decoding boosts speed, variants such as Medusa and EAGLE, chunked prefill and prefill/decode disaggregation, the latency versus throughput trade-off, and
2026-06-26 · 13 min read #speculative-decoding#throughput#inference#mlops#latencyPrompt Caching for Agent Apps: A Practical Guide to Lower Cost and Latency
A practical guide to prompt caching for agent applications, including why it matters, how OpenAI and Anthropic differ, prompt structuring patterns, ROI scenarios, common mistakes, and a migration checklist.
2026-04-12 · 7 min read #prompt-caching#latency#cost-optimization#ai-agent#llmopsAccelerating LLM Inference 2-3x with Speculative Decoding: From Theory to Production
A deep dive into the mathematical foundations of Speculative Decoding, the Draft-Verify pipeline, acceptance probability analysis, practical implementation in vLLM/TensorRT-LLM, and Apple's Mirror Speculative Decoding.
2026-03-02 · 8 min read #llm#speculative-decoding#inference#optimization#vllm