Table of Contents
1. The Setup: A Healthy Dashboard Can Still Be Lying to You
Chapter 4 built real machinery for trust: per-rule breakdowns, trend lines, hard gates, soft thresholds. All of that machinery assumes the underlying eval score is an honest signal — that when it goes up, the agent actually got better at the real skill, and when it stays flat, nothing meaningfully changed. That assumption can break in ways that produce exactly the same green dashboard as genuine improvement. This chapter covers the two biggest ways it breaks — Goodhart's Law and overfitting to your own Golden Dataset — plus a handful of quieter, related pitfalls, and closes with the concrete practices that keep a benchmark honest as the agents being tested against it keep getting better.
2. Goodhart's Law
The law, in its most common phrasing (economist Charles Goodhart, 1975, later popularized in this wording by anthropologist Marilyn Strathern):
"When a measure becomes a target, it ceases to be a good measure."
Any metric you track — an eval score, a sales quota, a test pass rate — is a proxy for something you actually care about but can't measure directly at scale. The proxy is useful only as long as moving the proxy and improving the real underlying thing stay correlated. The moment people (or a system) start optimizing specifically for the proxy, they'll find the cheapest way to move the number — and the cheapest way is essentially never "get genuinely better," because genuine improvement is hard. It's whatever narrow trick satisfies the metric's literal definition instead.
Two classic, non-AI examples, because seeing the pattern outside your own domain first makes it easier to spot inside it:
- The Soviet nail factory. Measure output by number of nails produced, and factories start producing huge quantities of tiny, useless nails. Switch the measure to weight, and factories start producing a few giant, equally useless slabs of iron. Neither metric was wrong on its own terms — both were satisfied — and neither produced usable nails.
- Teaching to the test. Once a standardized test score becomes the thing a school is measured and funded by, instruction can drift toward "whatever raises this specific score" rather than the broader learning the score was meant to stand in for.
Bring it home: an eval score is a proxy for "is this agent actually good at code and business-logic comprehension." The instant that score becomes the target — something an engineering team is specifically rewarded for moving, or baked into how an agent or model gets tuned — the score and the real skill can quietly decouple, exactly like the nail factory's count and the world's actual supply of usable nails.
3. Worked Example: Gaming the Woody's Hardware Eval
This is easiest to see with numbers, using the exact Golden Dataset from Chapter 3. Ten test cases, twenty rules each. Two deliberately degenerate "agents" — not real AI, just simple fixed strategies — show precisely how a single metric can be maximized without any real comprehension at all.
Agent X: "Flag all 20 rules as failing, every time"
This strategy never actually reads the application. It just always reports all 20 rules as failed.
| Case | Actual failing rules | Agent X flags | True positives | False positives | Precision |
|---|---|---|---|---|---|
| GD-001 | none (0) | all 20 | 0 | 20 | 0% |
| GD-002 | R1 (1) | all 20 | 1 | 19 | 5% |
| GD-006 | R11, R20 (2) | all 20 | 2 | 18 | 10% |
| GD-008 | R4, R12, R13 (3) | all 20 | 3 | 17 | 15% |
| (remaining 6 cases follow the same pattern) | |||||
Because Agent X flags every rule on every case, it can never miss a real failure — recall is a perfect 100%. It also gives the exact same output every time, so self-consistency is a perfect 100% too. And because 8 of the 10 golden cases are genuinely ineligible for some reason, its blanket "not eligible" verdict happens to be correct 8 times out of 10 — 80% overall verdict accuracy. Look at that on a dashboard that only tracks verdict accuracy, recall, and consistency, and Agent X looks like a strong, reliable performer. Precision is the only metric that tells the truth: averaged across all ten cases, it's 5.5% — nineteen times out of twenty, a "failure" this agent reports isn't real.
Agent Y: "Always say 'Eligible, no issues found'"
The mirror-image degenerate strategy: never flag anything, regardless of input.
| Metric | Agent X ("flag everything") | Agent Y ("always eligible") |
|---|---|---|
| Overall verdict accuracy | 80% | 20% |
| Rule-level recall | 100% | 0% |
| Rule-level precision | 5.5% | undefined — nothing ever flagged |
| Self-consistency (5 runs) | 100% | 100% |
Agent Y also scores a perfect 100% on self-consistency — it's not reasoning about the input at all, so of course it never disagrees with itself. If Chapter 4's dashboard leaned on consistency as a headline "reliability" signal, Agent Y would look like the most trustworthy agent in the fleet — while being wrong 8 times out of 10.
The general lesson
Every single metric this series has built — verdict accuracy, recall, precision, self-consistency — can individually be driven to a perfect score by a strategy with zero real comprehension. That's Goodhart's Law showing up concretely. The only real defense: track multiple, orthogonal metrics together, and get suspicious the moment one moves without the others. Recall at 100% with precision in single digits isn't a good agent — it's a red flag invisible on a recall-only dashboard and obvious the moment precision sits next to it.
4. Overfitting to Your Own Golden Dataset
Goodhart's Law is about incentives — a target distorting behavior. Overfitting is a related but distinct failure mode: it's about exposure. An agent (or the model/prompt behind it) can start doing unusually well on your specific golden cases not because it generalized the underlying rules, but because it has, in some sense, seen these exact cases before — through repeated prompt-tuning iterations, few-shot examples lifted from it, or the least visible version: because the dataset itself became part of a future model's training data.
This is not a hypothetical for this series. Chapter 3 published all ten Woody's Hardware golden cases in plain text, in a document now sitting on disk — and potentially the open web. If a future model is ever trained on text that includes that chapter, it can learn to answer "GD-008 fails R4, R12, and R13" as a memorized fact about this specific document rather than by genuinely reasoning through the rules. A real production benchmark would not publish its full golden set this openly; Chapter 3 did, deliberately, because the goal was teaching the mechanics of a Golden Dataset, not shipping a durable, contamination-resistant benchmark.
How this shows up in practice
- A score that's suspiciously perfect on the published, well-known cases but noticeably worse on freshly generated cases testing the same rules with different names, numbers, and phrasing.
- A discontinuous jump in score — say, 75% to 99% — with no corresponding change in model capability or agent architecture to explain it.
- Performance that doesn't transfer: an agent that aces GD-002 (age 17, fails R1) but stumbles on a same-rule variant with a different birth year and name wasn't applying the age rule — it was pattern-matching the specific example.
5. Other Pitfalls That Quietly Invalidate an Eval
| Pitfall | What happens | Why the dashboard still looks fine |
|---|---|---|
| Narrow / non-diverse Golden Dataset | All ten cases share similar structure, phrasing, and style. | The eval validates performance within that narrow style — real-world variety never gets tested. |
| Curator selection bias | The dataset builder unconsciously favors clean, easy-to-verify cases and avoids genuinely ambiguous ones. | The dataset systematically underrepresents the hardest real-world cases. |
| Correlated judge bias (LLM-as-judge) | The grading model shares blind spots with the agent being graded — especially if from the same model family. | Judge and agent can be wrong in the same way at the same time; a shared blind spot never triggers disagreement. |
| Stale ground truth | The business rules update; the Golden Dataset's expected outputs don't get updated to match. | The eval silently penalizes an agent for being right about new rules while grading it against outdated ones. |
| Metric myopia | The roadmap chases whatever moves the one dashboard number, at the expense of unmeasured dimensions (latency, cost, security). | The tracked metric climbs steadily — genuinely — while unmeasured quality erodes elsewhere. |
6. Keeping a Benchmark Honest Over Time
None of this means scores should never rise — genuine improvement is real and should show up as a rising score. The goal is narrower: being able to tell genuine improvement apart from gaming, memorization, or a stale/narrow benchmark, not suppressing improvement itself.
- Split "practice" from "exam," the way classical ML splits train/validation/test. Use one set of golden cases openly during development; keep a second, structurally similar but never-published set as the actual scored benchmark.
- Keep most of the dataset private. Publish only a small, representative sample for documentation and teaching.
- Rotate and regenerate test cases on a schedule — same rules, different names/numbers/phrasing, quarterly or faster.
- Never optimize, or even monitor, a single number. Report recall, precision, verdict accuracy, and consistency side by side, always.
- Investigate discontinuous jumps before celebrating them. A sudden, large improvement with no matching capability change is a memorization/gaming signature.
- Periodically test against fresh, never-seen cases alongside the known golden set. A gap between the two is the contamination signal.
- Version and diff the golden dataset like code. When a business rule changes, the golden cases need an explicit, tracked update.
- Add a few canary/decoy cases designed to resist memorization — structurally unusual scenarios that catch an agent coasting on memorized answers.
Treat the benchmark itself as a living, versioned product — not a fixed artifact built once and never touched again.
7. Key Takeaways
- Goodhart's Law — "when a measure becomes a target, it ceases to be a good measure" — applies directly to eval scores: the instant a single metric becomes the target, there's usually a cheap, degenerate way to max it out.
- The worked example proves it: an agent that blindly flags all 20 rules scores 100% recall, 100% consistency, and 80% verdict accuracy while being useless (5.5% precision); an agent that always says "eligible" scores 100% consistency while being wrong 80% of the time.
- The defense: track multiple, orthogonal metrics together and treat any one metric moving without the others as a reason to investigate, not celebrate.
- Overfitting/memorization is a distinct risk from Goodhart's Law — it's about exposure, not incentive. A published Golden Dataset is itself a contamination risk for its own future scoring.
- Other quiet failure modes: too-narrow dataset, curator selection bias, correlated judge bias, stale ground truth, and metric myopia.
- Keeping a benchmark honest is an ongoing practice: practice/exam splits, mostly-private data, periodic rotation, multi-metric reporting, investigating jumps, fresh-case spot checks, versioning, and canary cases.
- The goal is never to suppress rising scores — it's to tell genuine improvement apart from gaming or contamination when a score does rise.
8. Glossary
| Term | Meaning |
|---|---|
| Goodhart's Law | "When a measure becomes a target, it ceases to be a good measure" — optimizing directly for a proxy metric tends to decouple it from the real quality it represents. |
| Overfitting (to a golden dataset) | Performing well on specific known test cases through memorization or repeated exposure rather than genuine, generalizable capability. |
| Contamination | A test set (or answers derived from it) ending up inside a model's training data, invalidating it as an independent measure of capability. |
| Practice / exam split | Keeping one dataset visible for development and tuning, and a separate, never-seen dataset for the actual trusted score. |
| Canary / decoy case | A test case deliberately designed to be resistant to memorization, included to catch an agent coasting on memorized answers. |
| Metric myopia | Letting a roadmap optimize for whatever a current dashboard happens to measure, at the expense of real but unmeasured quality dimensions. |
| Discontinuous jump | A sudden, large eval score improvement with no matching capability or architecture change — a common signature of gaming or contamination. |
This is Chapter 5 in the Eval Basics series. Next: how many times do you actually need to run an eval before you can trust the number? Sample sizes, confidence intervals, and the real cost/statistical-power tradeoff of running evals at scale — plus the ten specific, named ways an autonomous agent actually fails. — continue to Chapter 6: The Ten Agent Fault Lines →