Evals vs. Classic QA — and Building a Scorecard a Team Actually Trusts

Technique 1 — Parameterize over the Golden Dataset

Pytest's @pytest.mark.parametrize and JUnit 5's @ParameterizedTest were built to run one test body across many inputs — exactly what a Golden Dataset needs.

import pytest
from golden_dataset import GOLDEN_CASES          # the 10 entries from Chapter 3
from agent_client import ask_agent_to_validate
from eval_metrics import rule_level_recall, rule_level_precision, agreement_rate

@pytest.mark.parametrize("case", GOLDEN_CASES, ids=lambda c: c["id"])
def test_agent_matches_ground_truth(case, eval_report):
    runs = [ask_agent_to_validate(case["input"]) for _ in range(5)]   # 5 samples per case

    avg_recall = sum(rule_level_recall(r, case["expected"]) for r in runs) / len(runs)
    avg_precision = sum(rule_level_precision(r, case["expected"]) for r in runs) / len(runs)
    consistency = agreement_rate(runs)

    eval_report.record(case["id"], recall=avg_recall, precision=avg_precision, consistency=consistency)

    assert avg_recall >= 0.90, f"{case['id']}: recall {avg_recall:.0%} below threshold"
    assert consistency >= 0.80, f"{case['id']}: consistency {consistency:.0%} below threshold"

Technique 2 — Repeat runs to measure self-consistency

JUnit 5's @RepeatedTest was designed to help surface flaky tests. It turns out to be a near-perfect fit for measuring an eval's self-consistency instead.

@RepeatedTest(5)
void agentFindsKnownBankruptcyFailure(RepetitionInfo repInfo) {
    AgentResult result = agentClient.validate(GD_004_INPUT);
    consistencyTracker.record("GD-004", repInfo.getCurrentRepetition(), result);

    assertTrue(result.failedRules().contains("R10"),
        "Run " + repInfo.getCurrentRepetition() + " missed the bankruptcy rule (R10)");
}

@AfterAll
static void checkConsistency() {
    double agreement = consistencyTracker.agreementRateFor("GD-004");
    assertTrue(agreement >= 0.80, "GD-004 consistency " + agreement + " was below the 80% threshold");
}

Technique 3 — Assert on a threshold, not an exact value

assertEquals(expected, actual) fails the instant one value doesn't match. Evals instead accumulate a score across many runs and assert that the aggregate clears a bar — closer to assertTrue(recall >= 0.90) than to exact equality. Pytest's pytest.approx and JUnit's soft-assertion patterns are useful low-level primitives here, but the real shift is conceptual: you're asserting on a statistic, not a value.

Technique 4 — Use fixtures/extensions for the bookkeeping evals specifically need

A Pytest fixture (or a JUnit TestWatcher extension) that automatically records the model version, temperature, prompt version, and timestamp alongside every result is exactly the kind of cross-cutting concern this tooling was built for — and it's what makes results reproducible later (Chapter 1).

Technique 5 — Keep the CI integration you already have

Because it's still fundamentally a JUnit or Pytest suite, it plugs into the CI pipeline, the JUnit-XML report format most dashboards already consume, and the failure-notification tooling your team already trusts — without standing up new infrastructure from scratch.

Setting thresholds

• Different rules deserve different bars. A rule tied to compliance or safety exposure — citizenship eligibility, consent, bankruptcy — deserves a much higher required recall than a cosmetic completeness check.
• Two tiers: hard gates and soft thresholds. A hard gate automatically blocks a merge/release when a critical metric falls below its bar; a soft threshold flags a metric for human review without blocking anything.
• Check against the noise floor before calling something a regression. A 3-point drop might be real or might be well within normal run-to-run spread — you need to know your metric's typical noise level first.
• This is structurally closer to an SLO than to a pass/fail gate. Site reliability engineering doesn't demand 100% uptime; it sets a target and an error budget, then treats sustained drift as the trigger for action.