Tag: #llm-as-judge
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 5 posts
5 posts are tagged #llm-as-judge; the most recent was published on 2026-08-12.
Often tagged together: #evaluation 4 #llm 3 #2026-03 1 #AI에이전트 1 #agent 1
Most read in the last 90 days:
The Evaluator Bottleneck — A Weak Grader Caps the Whole System
If the score does not move no matter how much you fix the harness, the bottleneck may be the evaluator, not the harness. You cannot select for a quality you cannot measure, which is why a weak grader becomes the ceiling
2026-08-12 · 5 min read #llm#agent#harness-engineering#하네스엔지니어링#AI에이전트In Eval-Driven Development, the First Thing to Calibrate Is the Judge
The eval-driven development retrospective Airbnb Engineering published in July 2026 is less a plea to write the eval set first than a plea to earn the right to treat the grading model as an instrument. This post lays out
2026-08-09 · 9 min read #ai#llm#eval-driven-development#llm-as-judge#evaluationHow Text, Images, and Agents Are Each Measured — Why the Three Domains Measure Fundamentally Different Things ♪ Listenable
Text, images, and agents all use the word "performance," but their measurement structures are entirely different. Text splits into multiple-choice that pretends to have a correct answer and open-ended generation that has
2026-08-02 · 21 min read #llm-evaluation#multimodal#llm-as-judge#agent-benchmark#metricsHow to Stop Evaluating LLMs on Vibes — Sample Size, Judge Bias, and CI Regression Tests
A team that edits a prompt and ships on "it seems better now" has no way to see the regressions quietly piling up. This post splits evaluation into four layers — assertions, golden datasets, LLM-as-judge, and human revie
2026-07-26 · 12 min read #llm#evaluation#llm-as-judge#statistics#regression-testingBuilding a Chatbot Evaluation Framework: LLM-as-Judge, RAGAS, and Automated Testing Pipelines ♪ Listenable
A comprehensive guide to building systematic quality evaluation for LLM-based chatbots. Covers RAG pipeline evaluation with RAGAS, LLM-as-Judge patterns, automated testing pipelines, and production monitoring.
2026-03-10 · 27 min read #chatbot#evaluation#ragas#llm-as-judge#testing