Tag: #lm-evaluation-harness
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 3 posts
Taking Apart LLM Benchmark Tooling — Why the Same MMLU Gives Different Scores in Different Harnesses ♪ Listenable
A benchmark score is not a property of the model — it is the result of a measurement performed under specific conditions. The incident where the same LLaMA 65B scored 63.6 and 48.8 on the same MMLU at the same time shows
2026-08-02 · 21 min read #llm-evaluation#benchmark#lm-evaluation-harness#helm#reproducibilityAI Safety, Evals and Red-Teaming in 2026 — Deep Dive into Inspect AI, Garak, PyRIT, Promptfoo, OpenAI Evals, lm-eval-harness ♪ Listenable
A single-page map of the 2026 AI safety, evaluation, and red-teaming ecosystem. Inspect AI (Anthropic, adopted by UK AISI), Garak (NVIDIA then independent), PyRIT (Microsoft), Promptfoo (YC), OpenAI Evals, lm-evaluation-
2026-05-16 · 22 min read #ai-safety#red-teaming#evaluation#inspect-ai#garakLLM Evaluation and Benchmarking Guide: Measuring What Actually Matters
A comprehensive guide to LLM evaluation and benchmarking. Covers standard academic benchmarks (MMLU, HELM, MT-Bench), production evaluation pipelines, LLM-as-judge, human evaluation, RAG evaluation, safety testing, and b
2026-03-17 · 18 min read #llm#evaluation#benchmark#mmlu#helm