Tag: #llm-evaluation
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 7 posts
7 posts are tagged #llm-evaluation; the most recent was published on 2026-08-14.
Often tagged together: #helm 2 #swe-bench 2 #agent-benchmark 1 #agentbench 1 #agieval 1
Most read in the last 90 days:
- AI Agent & LLM Benchmarks 2026 — SWE-bench Verified / ARC-AGI 2 / GAIA / MMLU-Pro / GPQA / LiveCodeBench / Chatbot Arena Deep Dive
- Taking Apart LLM Benchmark Tooling — Why the Same MMLU Gives Different Scores in Different Harnesses
- After You Have Collected It — Langfuse Dashboards, the Metrics API, and Scores on Traces
After You Have Collected It — Langfuse Dashboards, the Metrics API, and Scores on Traces
Collecting traces and getting answers out of them are different jobs. This post lays out which metric axes Langfuse offers and which dimensions you have to slice them by, then walks the Metrics API v2 query structure as
2026-08-14 · 9 min read #observability#langfuse#llm-evaluation#metrics-api#dashboardTaking Apart LLM Benchmark Tooling — Why the Same MMLU Gives Different Scores in Different Harnesses ♪ Listenable
A benchmark score is not a property of the model — it is the result of a measurement performed under specific conditions. The incident where the same LLaMA 65B scored 63.6 and 48.8 on the same MMLU at the same time shows
2026-08-02 · 21 min read #llm-evaluation#benchmark#lm-evaluation-harness#helm#reproducibilityBuilding an Eval Set for Your Own Service — From Traffic Collection to Statistical Significance ♪ Listenable
Public benchmarks cannot measure your problem for you, because the input distribution, the definition of a correct answer, and the cost structure of failure are all different. This post walks through, with real code, how
2026-08-02 · 23 min read #llm-evaluation#eval-set#rubric#statistics#regression-testingHow Text, Images, and Agents Are Each Measured — Why the Three Domains Measure Fundamentally Different Things ♪ Listenable
Text, images, and agents all use the word "performance," but their measurement structures are entirely different. Text splits into multiple-choice that pretends to have a correct answer and open-ended generation that has
2026-08-02 · 21 min read #llm-evaluation#multimodal#llm-as-judge#agent-benchmark#metricsSeparating Signal from Noise When You Evaluate AI Coding Models — Why SWE-bench Got Shaky
OpenAI's evals team argues that SWE-bench Verified, the most widely used coding benchmark, no longer gives meaningful signal because of contamination and design flaws. Look at benchmarks through two axes — signal (the po
2026-07-11 · 6 min read #llm-evaluation#coding-benchmarks#swe-bench#benchmarks#ai-codingAI Agent & LLM Benchmarks 2026 — SWE-bench Verified / ARC-AGI 2 / GAIA / MMLU-Pro / GPQA / LiveCodeBench / Chatbot Arena Deep Dive
A single-page map of the 30+ AI benchmarks that matter in 2026. From SWE-bench / SWE-bench Verified / SWE-bench Multimodal to AgentBench, WebArena and GAIA, ARC-AGI 2 (Chollet $1M prize), RE-Bench (METR), Frontier Math (
2026-05-16 · 19 min read #ai-benchmark#llm-evaluation#swe-bench#swe-bench-verified#agentbenchRAGAS Complete Guide: How to Quantitatively Evaluate Your RAG System
Learn to use RAGAS — the gold standard for RAG evaluation — to measure Faithfulness, Answer Relevancy, Context Precision, and Context Recall. Includes complete Python code, automated CI/CD evaluation pipelines, synthetic
2026-03-18 · 9 min read #ragas#rag-evaluation#llm-evaluation#ai-development#quality