Tag: #evaluation

← All posts

• English for AI Engineering Meetings: Saying You Do Not Know Without Losing Credibility ♪ Listenable

  The English an AI team meeting needs most is not the language of certainty but the language that makes uncertainty credible. Reporting eval results together with their limits, flagging a regression whose cause is still u

  2026-08-16 · 31 min read #english#business-english#meeting#ai#llm

• The Evaluator Bottleneck — A Weak Grader Caps the Whole System

  If the score does not move no matter how much you fix the harness, the bottleneck may be the evaluator, not the harness. You cannot select for a quality you cannot measure, which is why a weak grader becomes the ceiling

  2026-08-12 · 5 min read #llm#agent#harness-engineering#하네스엔지니어링#AI에이전트

• Reward Hacking — The Metric Rises While the Task Fails

  If the cheapest way for an agent to pass the tests is to edit the tests, the agent will edit the tests. Reward hacking is not a bug; it is the exact optimization of the goal we wrote down. Part 6 of the harness engineeri

  2026-08-12 · 5 min read #llm#agent#harness-engineering#하네스엔지니어링#AI에이전트

• Video Generation and Understanding Technical Reports: What to Read, and Why Constraints Beat Demos

  Ten video generation and understanding technical reports, each verified by opening the arXiv abstract page directly. CogVideoX, Movie Gen, HunyuanVideo, LTX-Video, Wan, Seedance 1.0 and 2.0, plus Qwen2.5-VL, VideoLLaMA 3

  2026-08-12 · 7 min read #ai-papers#paper-review#technical-report#video-generation#diffusion-transformer

• Text LLM Technical Reports: What to Read, and How to Read Design Decisions Instead of Rankings

  Nine text-LLM technical reports, each verified by opening the arXiv abstract page directly. What DeepSeek-V3, DeepSeek-R1, Qwen3, Gemma 3, Olmo 3, Kimi K2, MiniMax-01, Mellum2 and s1 each did that was new, and which limi

  2026-08-12 · 7 min read #ai-papers#paper-review#technical-report#llm#moe

• OCR and Document Understanding Technical Reports: What to Read, and Why Parsing Is Not Finished

  Ten OCR and document understanding technical reports, each verified by opening the arXiv abstract page directly. From Donut and Nougat through GOT-OCR2.0, olmOCR, DeepSeek-OCR and its successor, GLM-OCR, Qianfan-OCR, Min

  2026-08-12 · 7 min read #ai-papers#paper-review#technical-report#ocr#document-ai

• How to Read Leaderboards and Benchmarks: Why SOTA Has Such a Short Shelf Life

  Twelve benchmark methodology papers, each verified by opening the arXiv abstract page directly, assembled into a guide for reading leaderboard numbers. Data contamination, prompt format sensitivity, eval harness differen

  2026-08-12 · 7 min read #ai-papers#paper-review#technical-report#benchmark#evaluation

• Reading a Technical Report Critically — What Gets Written Down and What Goes Missing

  How to separate the verifiable from the unverifiable in a model technical report. Covers the limits of self-reported benchmarks, the places where a config and a report disagree, values that could not be read from gated r

  2026-08-12 · 7 min read #ai-papers#model-internals#tech-report#benchmarks#evaluation

• Speech Recognition and Synthesis Technical Reports: What to Read, and What a Single WER Hides

  Ten speech recognition and synthesis technical reports, each verified by opening the arXiv abstract page directly. Whisper, Omnilingual ASR, Qwen3-ASR, the Open ASR Leaderboard, Seed-TTS and F5-TTS, CosyVoice 2, Qwen3-TT

  2026-08-12 · 7 min read #ai-papers#paper-review#technical-report#speech-recognition#text-to-speech

• Image Generation and Understanding Technical Reports: What to Read, and Why Design Beats Sample Images

  Eleven image generation and understanding technical reports, each verified by opening the arXiv abstract page directly. Rectified flow transformers, VAR, Emu3, SANA, Janus-Pro, FLUX.1 Kontext, the Qwen-Image line, Seedre

  2026-08-12 · 8 min read #ai-papers#paper-review#technical-report#image-generation#diffusion-transformer

• A Harness Is Not Configuration but a Deployable — The Real Bottleneck of the Self-Improvement Loop

  The harness engineering post Lilian Weng published in July 2026 gives a name to the whole system wrapped around a model and treats it as a single engineering object. Rather than restating that definition, this post cover

  2026-08-09 · 7 min read #llm#agent#harness-engineering#context-engineering#evaluation

• In Eval-Driven Development, the First Thing to Calibrate Is the Judge

  The eval-driven development retrospective Airbnb Engineering published in July 2026 is less a plea to write the eval set first than a plea to earn the right to treat the grading model as an instrument. This post lays out

  2026-08-09 · 9 min read #ai#llm#eval-driven-development#llm-as-judge#evaluation

• What LLM Ops Actually Does — Reproducibility, Contamination, Checkpoints, Promotion, and Rollback

  This post organizes LLM Ops as a list of responsibilities, not a list of tools. It covers what belongs in a run manifest that lets you reconstruct a training run, how to prevent and audit eval-set contamination, the form

  2026-08-02 · 14 min read #mlops#llmops#reproducibility#evaluation#model-registry

• Gemini Robotics 2 and the Robot Foundation Model — What Whole-Body Control Actually Changes

  On July 30, 2026, Google DeepMind unveiled Gemini Robotics 2, announcing that a single vision-language-action model now controls a humanoid from its toes to its fingertips. Interestingly, the numbers released alongside i

  2026-07-31 · 14 min read #ai#robotics#vla#foundation-models#evaluation

• When RAG Answers the Wrong Thing — a Debugging Procedure That Separates Retrieval Failure from Generation Failure

  When RAG returns a wrong answer, most teams start by fixing the prompt, but the majority of real causes sit in the retrieval stage. This post starts with the single experiment — hand the model the gold chunk directly — t

  2026-07-26 · 13 min read #llm#rag#retrieval#chunking#evaluation

• How to Stop Evaluating LLMs on Vibes — Sample Size, Judge Bias, and CI Regression Tests

  A team that edits a prompt and ships on "it seems better now" has no way to see the regressions quietly piling up. This post splits evaluation into four layers — assertions, golden datasets, LLM-as-judge, and human revie

  2026-07-26 · 12 min read #llm#evaluation#llm-as-judge#statistics#regression-testing

• Is AI Code Review Actually Worth It — What Measured Evidence Says About Accuracy and False Positives ♪ Listenable

  Marketing copy for AI code review tools is full of numbers like "80% of PRs need no human comment," yet almost none of it publishes precision and false-positive rates alongside. Pull the public measurements together and

  2026-07-17 · 25 min read #ai#code-review#static-analysis#evaluation#software-engineering

• Simulated Customers Never Walk Away — Where LLM User Simulators Inflate Agent Scores ♪ Listenable

  In τ-bench-style conversational agent benchmarks, the 'user' role is played by another LLM. But this simulator isn't the thing being measured — it's the measuring instrument, and instruments need calibration. Three verif

  2026-07-16 · 21 min read #ai#llm#evaluation#agents#simulation

• What a Good Agent Benchmark Looks Like in 2026 — UniClawBench, Live Containers, and a Hidden Supervisor

  UniClawBench, posted to arXiv in July 2026 by HKU MMLab, is a self-described capability-driven benchmark for proactive agents. Instead of matching against static, pre-recorded answers, it runs agents inside live Docker c

  2026-07-11 · 5 min read #ai#agents#evaluation#benchmark#llm

• tts-bench: comparing local TTS models when quality is subjective

  tts-bench is a local benchmark by 5uck1ess for comparing 55 text-to-speech models on hardware you own. It splits evaluation into three lenses: Speed (TTFA, RTF, memory), Listen (every model on every prompt, judged by ear

  2026-07-11 · 5 min read #tts#text-to-speech#benchmark#local-ai#evaluation