Tag: #benchmark
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 16 posts
16 posts are tagged #benchmark; the most recent was published on 2026-08-14.
Often tagged together: #llm 8 #ai 7 #evaluation 7 #2026-03 6 #open-source 3
Most read in the last 90 days:
Gemini 3.7 Flash, Its Introductory Price and Its Three-Week Cadence — Why Model Cost Is a Conditional Value, Not a Fixed One
What a practitioner should take from the Gemini 3.7 Flash announcement is not the size of the benchmark gains but two other things: an introductory price that doubles on a named date, and the fact that the previous model
2026-08-14 · 7 min read #llm#cost-optimization#benchmark#api-design#capacity-planningHow to Read Leaderboards and Benchmarks: Why SOTA Has Such a Short Shelf Life
Twelve benchmark methodology papers, each verified by opening the arXiv abstract page directly, assembled into a guide for reading leaderboard numbers. Data contamination, prompt format sensitivity, eval harness differen
2026-08-12 · 7 min read #ai-papers#paper-review#technical-report#benchmark#evaluationA Single Instruction Can Take 62 Seconds — Latency Is a Property of the Path, Not of the Instruction
The Assembly Hall of Shame is a leaderboard for the competition to make a single instruction as slow as possible. At the bottom, nop takes 1 cycle; at the top, fxrstor64 takes 198 billion cycles, or 62 seconds. Read that
2026-08-09 · 11 min read #os-concepts#performance#cpu#microarchitecture#benchmarkReasoning Effort Is Not a Model Choice but a Per-Request Deployment Parameter
The DeepSeek V4 Flash 0731 results page published by ARC Prize carries not one score but three, one per reasoning effort level. This post computes what can actually be read out of those three numbers: that the same step
2026-08-09 · 8 min read #llm#benchmark#arc-agi#inference#costMaking vLLM Fast — Configuration, Internals, and Where to Actually Touch the Code ♪ Listenable
A step-by-step walk through improving vLLM performance, starting from everything you can fix without touching code. Covers batching-related arguments, prefix caching, chunked prefill, and quantization choice first, then
2026-08-02 · 21 min read #vllm#llm-inference#paged-attention#benchmark#schedulerTaking Apart LLM Benchmark Tooling — Why the Same MMLU Gives Different Scores in Different Harnesses ♪ Listenable
A benchmark score is not a property of the model — it is the result of a measurement performed under specific conditions. The incident where the same LLaMA 65B scored 63.6 and 48.8 on the same MMLU at the same time shows
2026-08-02 · 21 min read #llm-evaluation#benchmark#lm-evaluation-harness#helm#reproducibilityBrowser and Computer-Use Agents: Where They Actually Stand, and What the Benchmarks Really Measure ♪ Listenable
"A computer-use agent hit 83.5% on OSWorld" and "even the strongest agent finishes only 20.6%" are both facts published in 2026, and both are true. The first is OSWorld 1.0; the second is OSWorld 2.0 from the same team.
2026-07-17 · 24 min read #ai#computer-use#browser-agents#benchmark#prompt-injectionHow AI Agent Memory Is Actually Built — Four Designs, and What the Benchmarks Really Prove ♪ Listenable
"Agent memory" is not a single technique but a blur of at least four distinct designs — a file scratchpad, summarization/compaction, vector retrieval, and knowledge graphs. This post pins down what each actually does fro
2026-07-17 · 21 min read #ai#ai-agent#agent-memory#llm#benchmarkWhat a Good Agent Benchmark Looks Like in 2026 — UniClawBench, Live Containers, and a Hidden Supervisor
UniClawBench, posted to arXiv in July 2026 by HKU MMLab, is a self-described capability-driven benchmark for proactive agents. Instead of matching against static, pre-recorded answers, it runs agents inside live Docker c
2026-07-11 · 5 min read #ai#agents#evaluation#benchmark#llmtts-bench: comparing local TTS models when quality is subjective
tts-bench is a local benchmark by 5uck1ess for comparing 55 text-to-speech models on hardware you own. It splits evaluation into three lenses: Speed (TTFA, RTF, memory), Listen (every model on every prompt, judged by ear
2026-07-11 · 5 min read #tts#text-to-speech#benchmark#local-ai#evaluationBFCL Benchmark Complete Guide 2025: Tool Calling Evaluation, Leaderboard Analysis, Model Comparison
Everything about BFCL (Berkeley Function Calling Leaderboard)! Benchmark categories (Simple/Multiple/Parallel/Relevance/AST), evaluation metrics, model performance comparison (Claude/GPT/Gemini/Llama), running your own e
2026-03-25 · 22 min read #bfcl#benchmark#tool-calling#function-calling#evaluation2025 Open Source AI Models Showdown: DeepSeek R1 vs Llama 4 vs Qwen 3 vs Mistral ♪ Listenable
DeepSeek R1 (671B/37B), Llama 4 Scout/Maverick, Qwen 3 (235B MoE), Mistral 8x22B — complete comparison of the 2025 open-source AI model leaders with benchmarks, licenses, deployment guides, and cost analysis.
2026-03-22 · 20 min read #open-source#ai#llm#deepseek#llamaMarch 2025 Tech·AI·K-POP Weekly Digest: From GTC to BTS Comeback
A comprehensive roundup of March 2025 highlights: NVIDIA GTC Blackwell Ultra announcement, Gemini 2.5 Pro topping benchmarks, MCP becoming the industry standard, DeepSeek-R1 open-source shock, BTS full-group comeback aft
2026-03-21 · 12 min read #culture#ai#kpop#nvidia#gtcComplete Guide to LLM Evaluation and Benchmarking: MMLU, MT-Bench, RAGAS, LM-Eval
A complete guide to evaluating LLMs correctly. Covers standard benchmarks such as MMLU, MT-Bench, and HumanEval, evaluating RAG systems with RAGAS, practical use of LM-Evaluation-Harness, and building production LLM eval
2026-03-17 · 17 min read #llm#evaluation#benchmark#ragas#lm-evalLLM Evaluation and Benchmarking Guide: Measuring What Actually Matters
A comprehensive guide to LLM evaluation and benchmarking. Covers standard academic benchmarks (MMLU, HELM, MT-Bench), production evaluation pipelines, LLM-as-judge, human evaluation, RAG evaluation, safety testing, and b
2026-03-17 · 18 min read #llm#evaluation#benchmark#mmlu#helmAI Benchmark Datasets Complete Guide: ImageNet, COCO, GLUE, MMLU, HumanEval ♪ Listenable
A complete guide to key benchmark datasets for AI model evaluation. Detailed analysis of computer vision (ImageNet, COCO, ADE20K), NLP (GLUE, SuperGLUE, SQuAD, MMLU), code (HumanEval, MBPP), and LLM evaluation (HELM, MT-
2026-03-17 · 25 min read #benchmark#datasets#imagenet#coco#glue