Chapter 3 — Building a Golden Dataset (Worked Example)

Blue — core concept

Red — the problem / pitfall

Green — takeaways & fixes

Yellow — definitions & highlights

← Back to All Chapters

1The App We're Testing

2The 20 Business Rules

3The Rule Engine

4Building the Golden Dataset

5Using the Golden Dataset as an Eval

6Key Takeaways

7Glossary

1. The App We're Testing: Woody's Hardware Credit Application Validator

Woody's Hardware is a regional hardware store chain that offers its own store credit card. Customers fill out an application (name, income, employment, existing debts, etc.), and the store needs to decide whether the application is eligible to move forward for processing — based on 20 business rules the credit team has defined. The web app itself is deliberately simple: a single HTML page (Bootstrap for styling, vanilla JavaScript for logic, no build step, no framework) where an employee uploads the applicant's JSON file and clicks Validate.

index.html

<!doctype html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Woody's Hardware — Credit Application Validator</title>
  <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" rel="stylesheet">
</head>
<body class="bg-light">
  <div class="container py-5">
    <h1 class="mb-4">Woody's Hardware — Credit Application Validator</h1>
    <div class="card p-4 shadow-sm">
      <input type="file" id="fileInput" accept=".json" class="form-control mb-3">
      <button id="validateBtn" class="btn btn-primary">Validate</button>
    </div>
    <div id="result" class="mt-4"></div>
  </div>
  <script src="validator.js"></script>
  <script src="app.js"></script>
</body>
</html>

app.js — wires the button to the file reader and renders the result

app.js

document.getElementById("validateBtn").addEventListener("click", () => {
  const fileInput = document.getElementById("fileInput");
  const resultDiv = document.getElementById("result");

  if (!fileInput.files.length) {
    resultDiv.innerHTML = '<div class="alert alert-warning">Please choose a JSON file first.</div>';
    return;
  }

  const reader = new FileReader();
  reader.onload = () => {
    let application;
    try {
      application = JSON.parse(reader.result);
    } catch (err) {
      resultDiv.innerHTML = '<div class="alert alert-danger">That file is not valid JSON.</div>';
      return;
    }
    renderResult(validateApplication(application));
  };
  reader.readAsText(fileInput.files[0]);
});

function renderResult(result) {
  const resultDiv = document.getElementById("result");
  const banner = result.eligible
    ? '<div class="alert alert-success">Eligible — application can be processed.</div>'
    : '<div class="alert alert-danger">Not eligible — see failed rules below.</div>';

  const rows = result.ruleResults
    .map(
      (r) => `<tr class="${r.passed ? "table-success" : "table-danger"}">
        <td>${r.id}</td><td>${r.name}</td><td>${r.passed ? "Pass" : "Fail"}</td>
      </tr>`
    )
    .join("");

  resultDiv.innerHTML = `${banner}
    <table class="table table-bordered">
      <thead><tr><th>Rule</th><th>Description</th><th>Result</th></tr></thead>
      <tbody>${rows}</tbody>
    </table>`;
}

Everything that actually matters — the 20 rules — lives in validator.js, covered in the next section.

↑ Back to Table of Contents

2. The 20 Business Rules

ID	Rule	Requirement
R1	Minimum Age	Applicant must be at least 18 years old as of the application date.
R2	Valid SSN Format	SSN must match `XXX-XX-XXXX` and not be the placeholder `000-00-0000`.
R3	Valid Contact Info	Email must be a valid address format; phone must contain exactly 10 digits.
R4	Eligible Citizenship/Residency	Must be a US citizen or permanent resident (visa holders not eligible for this card).
R5	Operating State	Applicant's state must be one Woody's Hardware operates in: CA, TX, AZ, NV, OR, WA.
R6	Minimum Annual Income	Annual income must be at least $12,000.
R7	Eligible Employment Status	Status must be `employed`, `self_employed`, or `retired`.
R8	Debt-to-Income Ratio	Monthly debt payments must be ≤ 45% of monthly income.
R9	Reasonable Housing Payment	Monthly housing payment must be > $0 and ≤ 50% of monthly income.
R10	No Recent Bankruptcy	No bankruptcy filed in the last 7 years.
R11	Minimum Credit Score	Credit score must be provided and ≥ 620.
R12	Not an Existing Cardholder	Applicant must not already hold a Woody's Hardware store card.
R13	Credit Check Consent Given	Applicant must have explicitly consented to a credit check.
R14	Requested Limit In Range	Requested credit limit must be between $200 and $10,000.
R15	Requested Limit vs. Income	Requested credit limit must be ≤ 20% of annual income.
R16	Address Stability	Must have lived at current address ≥ 6 months (0.5 years).
R17	Valid ZIP Code Format	ZIP must be a valid 5-digit (or ZIP+4) US format.
R18	Open Credit Lines Cap	Number of open credit lines must be ≤ 15.
R19	Valid Name Fields	First and last name must be non-empty, letters/hyphens/apostrophes only.
R20	Application Completeness	Every required field (including nested address fields) must be present and non-null.

Eligibility is strict: an application is eligible only if all 20 rules pass. Any single failure makes it ineligible.

↑ Back to Table of Contents

3. The Rule Engine — Code That Implements the Rules

This is the ground truth. It's ordinary, deterministic JavaScript — no AI involved. That matters: the whole point of a Golden Dataset is that the "correct answer" comes from something you can trust completely, so you have something solid to grade an agent's guess against.

validator.js

// validator.js — Woody's Hardware credit application rule engine

const OPERATING_STATES = ["CA", "TX", "AZ", "NV", "OR", "WA"];

function computeAgeYears(dateOfBirthStr, asOfDateStr) {
  const dob = new Date(dateOfBirthStr);
  const asOf = new Date(asOfDateStr);
  let age = asOf.getFullYear() - dob.getFullYear();
  const monthDiff = asOf.getMonth() - dob.getMonth();
  if (monthDiff < 0 || (monthDiff === 0 && asOf.getDate() < dob.getDate())) {
    age--;
  }
  return age;
}

function digitsOnly(str) {
  return (str || "").replace(/\D/g, "");
}

const RULES = [
  { id: "R1", name: "Minimum Age",
    check: (app) => computeAgeYears(app.dateOfBirth, app.applicationDate) >= 18 },

  { id: "R2", name: "Valid SSN Format",
    check: (app) => /^\d{3}-\d{2}-\d{4}$/.test(app.ssn) && app.ssn !== "000-00-0000" },

  { id: "R3", name: "Valid Contact Info",
    check: (app) => /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(app.email || "") &&
                     digitsOnly(app.phone).length === 10 },

  { id: "R4", name: "Eligible Citizenship/Residency",
    check: (app) => ["US_CITIZEN", "PERMANENT_RESIDENT"].includes(app.citizenshipStatus) },

  { id: "R5", name: "Operating State",
    check: (app) => OPERATING_STATES.includes(app.address && app.address.state) },

  { id: "R6", name: "Minimum Annual Income",
    check: (app) => typeof app.annualIncome === "number" && app.annualIncome >= 12000 },

  { id: "R7", name: "Eligible Employment Status",
    check: (app) => ["employed", "self_employed", "retired"].includes(app.employmentStatus) },

  { id: "R8", name: "Debt-to-Income Ratio",
    check: (app) => {
      const monthlyIncome = app.annualIncome / 12;
      return monthlyIncome > 0 && app.monthlyDebtPayments / monthlyIncome <= 0.45;
    } },

  { id: "R9", name: "Reasonable Housing Payment",
    check: (app) => {
      const monthlyIncome = app.annualIncome / 12;
      return app.monthlyHousingPayment > 0 && monthlyIncome > 0 &&
             app.monthlyHousingPayment / monthlyIncome <= 0.5;
    } },

  { id: "R10", name: "No Bankruptcy in Last 7 Years",
    check: (app) => app.bankruptcyLast7Years === false },

  { id: "R11", name: "Minimum Credit Score",
    check: (app) => typeof app.creditScore === "number" && app.creditScore >= 620 },

  { id: "R12", name: "Not an Existing Cardholder",
    check: (app) => app.existingStoreCardholder === false },

  { id: "R13", name: "Credit Check Consent Given",
    check: (app) => app.consentToCreditCheck === true },

  { id: "R14", name: "Requested Limit In Range ($200–$10,000)",
    check: (app) => app.requestedCreditLimit >= 200 && app.requestedCreditLimit <= 10000 },

  { id: "R15", name: "Requested Limit vs. Income (≤ 20%)",
    check: (app) => app.requestedCreditLimit <= 0.2 * app.annualIncome },

  { id: "R16", name: "Address Stability (≥ 6 months)",
    check: (app) => app.yearsAtAddress >= 0.5 },

  { id: "R17", name: "Valid ZIP Code Format",
    check: (app) => /^\d{5}(-\d{4})?$/.test((app.address && app.address.zip) || "") },

  { id: "R18", name: "Open Credit Lines ≤ 15",
    check: (app) => app.numberOfOpenCreditLines <= 15 },

  { id: "R19", name: "Valid Name Fields",
    check: (app) => /^[A-Za-z'-]+$/.test(app.firstName || "") &&
                     /^[A-Za-z'-]+$/.test(app.lastName || "") },

  { id: "R20", name: "Application Completeness",
    check: (app) => {
      const required = [
        "applicantId", "firstName", "lastName", "dateOfBirth", "ssn", "email", "phone",
        "yearsAtAddress", "employmentStatus", "annualIncome", "monthlyHousingPayment",
        "requestedCreditLimit", "existingStoreCardholder", "bankruptcyLast7Years",
        "creditScore", "numberOfOpenCreditLines", "monthlyDebtPayments",
        "citizenshipStatus", "consentToCreditCheck", "applicationDate"
      ];
      const hasTopLevel = required.every((f) => app[f] !== undefined && app[f] !== null);
      const addr = app.address;
      const hasAddress = Boolean(addr && addr.street && addr.city && addr.state && addr.zip);
      return hasTopLevel && hasAddress;
    } }
];

function validateApplication(app) {
  const ruleResults = RULES.map((rule) => {
    let passed;
    try {
      passed = Boolean(rule.check(app));
    } catch (err) {
      passed = false; // a rule that can't even be evaluated counts as failed, not as a crash
    }
    return { id: rule.id, name: rule.name, passed };
  });

  const failedRules = ruleResults.filter((r) => !r.passed).map((r) => r.id);
  return { eligible: failedRules.length === 0, failedRules, ruleResults };
}

One detail worth pausing on: applicationDate. Rule R1 doesn't compute age against "today" — it computes age against a fixed applicationDate field carried inside the JSON itself. This is deliberate, and it's the same lesson from Chapter 1 applied to dataset design: if age were computed against the real current date, the "correct" answer for a test case would silently drift every year. Anything time-dependent in a golden test case has to be pinned, not left to the clock.

↑ Back to Table of Contents

4. Building the Golden Dataset

A Golden Dataset entry is a triple: (synthetic input, the actual verified output, and what the entry is testing). The "actual verified output" here isn't guessed or written by hand — it's produced by actually running validateApplication(), which is exactly what makes it trustworthy ground truth.

ID	Scenario	Rule(s) under test	Expected `eligible`	Expected `failedRules`
GD-001	Fully eligible baseline	—	`true`	`[]`
GD-002	Applicant is 17 years old	R1	`false`	`["R1"]`
GD-003	Address in a non-operating state (NY)	R5	`false`	`["R5"]`
GD-004	Bankruptcy filed within the last 7 years	R10	`false`	`["R10"]`
GD-005	Debt-to-income ratio of 60%	R8	`false`	`["R8"]`
GD-006	Credit score missing (`null`)	R11, R20	`false`	`["R11","R20"]`
GD-007	Every numeric field sits exactly on its boundary	R1,R6,R8,R9,R11,R14,R16,R18	`true`	`[]`
GD-008	Visa holder, existing cardholder, no consent	R4, R12, R13	`false`	`["R4","R12","R13"]`
GD-009	Credit score one point below minimum (619)	R11	`false`	`["R11"]`
GD-010	SSN missing its dashes (malformed format)	R2	`false`	`["R2"]`

Note the mix: single-rule isolations (GD-002 through GD-010 mostly change exactly one thing), a fully-passing baseline (GD-001), a boundary case where every field sits exactly on its pass/fail threshold (GD-007 — this is the case most likely to expose an agent that silently treats >= as >), a cascading failure (GD-006, where one missing field trips two independent rules), and a multi-rule failure (GD-008, three unrelated rules failing at once).

The synthetic JSON test cases

All ten expected outputs were produced by actually running validateApplication() against each JSON payload below — not by manually guessing what the rules "should" produce. Expand any entry to see its full test data.

GD-001.json

{
  "applicantId": "APP-1001",
  "firstName": "John",
  "lastName": "Woodson",
  "dateOfBirth": "1990-06-10",
  "ssn": "123-45-6789",
  "email": "john.woodson@example.com",
  "phone": "(503) 555-1234",
  "address": { "street": "142 Pine St", "city": "Portland", "state": "OR", "zip": "97201" },
  "yearsAtAddress": 3.5,
  "employmentStatus": "employed",
  "employerName": "Cascade Manufacturing",
  "annualIncome": 52000,
  "monthlyHousingPayment": 1200,
  "requestedCreditLimit": 1500,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": false,
  "creditScore": 705,
  "numberOfOpenCreditLines": 4,
  "monthlyDebtPayments": 400,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

GD-002.json

{
  "applicantId": "APP-1002",
  "firstName": "Emma",
  "lastName": "Castillo",
  "dateOfBirth": "2008-03-01",
  "ssn": "234-56-7890",
  "email": "emma.castillo@example.com",
  "phone": "(619) 555-9832",
  "address": { "street": "88 Ocean Ave", "city": "San Diego", "state": "CA", "zip": "92101" },
  "yearsAtAddress": 1,
  "employmentStatus": "employed",
  "employerName": "Castillo Family Diner",
  "annualIncome": 15000,
  "monthlyHousingPayment": 300,
  "requestedCreditLimit": 500,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": false,
  "creditScore": 650,
  "numberOfOpenCreditLines": 2,
  "monthlyDebtPayments": 100,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

GD-003.json

{
  "applicantId": "APP-1003",
  "firstName": "Michael",
  "lastName": "Chen",
  "dateOfBirth": "1988-07-04",
  "ssn": "345-67-8901",
  "email": "michael.chen@example.com",
  "phone": "(212) 555-4477",
  "address": { "street": "500 5th Ave", "city": "New York", "state": "NY", "zip": "10110" },
  "yearsAtAddress": 2,
  "employmentStatus": "employed",
  "employerName": "Chen Consulting",
  "annualIncome": 40000,
  "monthlyHousingPayment": 900,
  "requestedCreditLimit": 2000,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": false,
  "creditScore": 680,
  "numberOfOpenCreditLines": 5,
  "monthlyDebtPayments": 500,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

GD-004.json

{
  "applicantId": "APP-1004",
  "firstName": "Sarah",
  "lastName": "Nguyen",
  "dateOfBirth": "1988-07-04",
  "ssn": "456-78-9012",
  "email": "sarah.nguyen@example.com",
  "phone": "(602) 555-7711",
  "address": { "street": "77 Desert Rd", "city": "Phoenix", "state": "AZ", "zip": "85001" },
  "yearsAtAddress": 2,
  "employmentStatus": "employed",
  "employerName": "Desert Valley Retail",
  "annualIncome": 40000,
  "monthlyHousingPayment": 900,
  "requestedCreditLimit": 2000,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": true,
  "creditScore": 680,
  "numberOfOpenCreditLines": 5,
  "monthlyDebtPayments": 500,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

GD-005.json

{
  "applicantId": "APP-1005",
  "firstName": "David",
  "lastName": "Alvarez",
  "dateOfBirth": "1988-07-04",
  "ssn": "567-89-0123",
  "email": "david.alvarez@example.com",
  "phone": "(702) 555-3300",
  "address": { "street": "12 Fremont St", "city": "Las Vegas", "state": "NV", "zip": "89101" },
  "yearsAtAddress": 2,
  "employmentStatus": "employed",
  "employerName": "Alvarez Trucking",
  "annualIncome": 30000,
  "monthlyHousingPayment": 600,
  "requestedCreditLimit": 1000,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": false,
  "creditScore": 680,
  "numberOfOpenCreditLines": 5,
  "monthlyDebtPayments": 1500,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

GD-006.json

{
  "applicantId": "APP-1006",
  "firstName": "Priya",
  "lastName": "Patel",
  "dateOfBirth": "1988-07-04",
  "ssn": "678-90-1234",
  "email": "priya.patel@example.com",
  "phone": "(206) 555-6620",
  "address": { "street": "900 Pike St", "city": "Seattle", "state": "WA", "zip": "98101" },
  "yearsAtAddress": 2,
  "employmentStatus": "employed",
  "employerName": "Pike Street Bakery",
  "annualIncome": 40000,
  "monthlyHousingPayment": 900,
  "requestedCreditLimit": 2000,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": false,
  "creditScore": null,
  "numberOfOpenCreditLines": 5,
  "monthlyDebtPayments": 500,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

GD-007.json

{
  "applicantId": "APP-1007",
  "firstName": "Alex",
  "lastName": "Rivera",
  "dateOfBirth": "2008-01-15",
  "ssn": "789-01-2345",
  "email": "alex.rivera@example.com",
  "phone": "(415) 555-8890",
  "address": { "street": "1 Boundary Ln", "city": "Austin", "state": "TX", "zip": "12345" },
  "yearsAtAddress": 0.5,
  "employmentStatus": "employed",
  "employerName": "Rivera Landscaping",
  "annualIncome": 12000,
  "monthlyHousingPayment": 500,
  "requestedCreditLimit": 200,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": false,
  "creditScore": 620,
  "numberOfOpenCreditLines": 15,
  "monthlyDebtPayments": 450,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

GD-008.json

{
  "applicantId": "APP-1008",
  "firstName": "Lars",
  "lastName": "Eriksen",
  "dateOfBirth": "1988-07-04",
  "ssn": "890-12-3456",
  "email": "lars.eriksen@example.com",
  "phone": "(213) 555-1120",
  "address": { "street": "44 Sunset Blvd", "city": "Los Angeles", "state": "CA", "zip": "90028" },
  "yearsAtAddress": 2,
  "employmentStatus": "employed",
  "employerName": "Eriksen Design Studio",
  "annualIncome": 40000,
  "monthlyHousingPayment": 900,
  "requestedCreditLimit": 2000,
  "existingStoreCardholder": true,
  "bankruptcyLast7Years": false,
  "creditScore": 680,
  "numberOfOpenCreditLines": 5,
  "monthlyDebtPayments": 500,
  "citizenshipStatus": "VISA_HOLDER",
  "consentToCreditCheck": false,
  "applicationDate": "2026-01-15"
}

GD-009.json

{
  "applicantId": "APP-1009",
  "firstName": "Grace",
  "lastName": "Kim",
  "dateOfBirth": "1988-07-04",
  "ssn": "901-23-4567",
  "email": "grace.kim@example.com",
  "phone": "(408) 555-2244",
  "address": { "street": "301 Market St", "city": "San Jose", "state": "CA", "zip": "95113" },
  "yearsAtAddress": 2,
  "employmentStatus": "employed",
  "employerName": "Kim & Co.",
  "annualIncome": 40000,
  "monthlyHousingPayment": 900,
  "requestedCreditLimit": 2000,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": false,
  "creditScore": 619,
  "numberOfOpenCreditLines": 5,
  "monthlyDebtPayments": 500,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

GD-010.json

{
  "applicantId": "APP-1010",
  "firstName": "Omar",
  "lastName": "Farouk",
  "dateOfBirth": "1988-07-04",
  "ssn": "123456789",
  "email": "omar.farouk@example.com",
  "phone": "(480) 555-7788",
  "address": { "street": "60 Cactus Way", "city": "Tempe", "state": "AZ", "zip": "85281" },
  "yearsAtAddress": 2,
  "employmentStatus": "employed",
  "employerName": "Farouk Hardware Supply",
  "annualIncome": 40000,
  "monthlyHousingPayment": 900,
  "requestedCreditLimit": 2000,
  "existingStoreCardholder": false,
  "bankruptcyLast7Years": false,
  "creditScore": 680,
  "numberOfOpenCreditLines": 5,
  "monthlyDebtPayments": 500,
  "citizenshipStatus": "US_CITIZEN",
  "consentToCreditCheck": true,
  "applicationDate": "2026-01-15"
}

↑ Back to Table of Contents

5. Using the Golden Dataset as an Eval

Now the payoff. Instead of asking "does this coding agent seem good at analyzing business logic?" — the unanswerable, gut-feel version — the Golden Dataset lets you ask something precise and repeatable: for each of these 10 applications, does the agent correctly predict what the code actually does?

Two ways to run this eval

Mode A — Code comprehension. Give the agent validator.js and one JSON application, and ask it to trace through the code and predict what validateApplication() returns. This tests whether the agent can correctly read and reason about code it didn't write.

Mode B — Business-rule application. Give the agent only the 20 rules in plain English and the JSON application — no source code at all — and ask it to apply the rules itself. This tests whether the agent can correctly reason about the business requirements before any code exists.

What to score — not just yes/no

A naive eval would only check the final eligible: true/false verdict. That's a mistake: a business needs to know not just whether an application was rejected, but which specific rules it failed — both to process it correctly and, in a regulated context like consumer credit, because U.S. lenders are generally required to give applicants specific, accurate reasons when an application is denied (adverse-action notices under the Equal Credit Opportunity Act).

So the eval scores at the rule level, not just the verdict level: overall eligibility accuracy, rule-level recall (of the rules that actually failed, how many did the agent identify?), rule-level precision (of the rules it claimed failed, how many actually did?), and consistency (run the same case multiple times — how much do the agent's own answers agree?).

Worked example: GD-008, run twice

GD-008 has three genuinely failing rules: R4 (visa holder), R12 (existing cardholder), R13 (no consent). Here's a plausible pair of runs from a Mode B eval — the exact "ask it twice" scenario from Chapter 1, now applied to this concrete business task.

Run 1 response: "This applicant is NOT eligible. Failing rules: R4 — citizenship status is VISA_HOLDER. R12 — existingStoreCardholder is true."

Run 2 response: "This applicant is NOT eligible. Failing rules: R4 — visa holder. R13 — consentToCreditCheck is false."

Metric	Run 1	Run 2
Eligibility verdict correct?	Yes	Yes
Rule-level recall (of R4, R12, R13)	2/3 = 66.7%	2/3 = 66.7%
Rule-level precision (no invented failures)	2/2 = 100%	2/2 = 100%
Rules found in common between the two runs	R4 only
Cross-run agreement (Jaccard: shared ÷ total unique rules found)	1 ÷ 3 = 33%

This is exactly why scoring only the top-level verdict would be dangerously misleading: both runs would show 100% accuracy on "did it correctly say not eligible," completely hiding that the agent is unreliable at the rule-level detail the business actually needs to act on correctly. The rule-level and consistency metrics catch what the verdict-only metric can't see.

Scaling this to a real scorecard

In practice, you run all 10 (or all 20-rule-isolated) Golden Dataset entries, multiple times each — say, 5 runs per entry, 50 runs total — and aggregate: overall accuracy across all verdicts, rule-level precision/recall across all 200 individual rule checks (10 entries × 20 rules), a consistency score (the average agreement rate per entry), and trend over time — rerunning the same Golden Dataset every time the agent, model, or prompt changes, and tracking whether these scores go up or down. This is what turns "we think the new model version is better" into a number you can actually point to.

↑ Back to Table of Contents

6. Key Takeaways

A Golden Dataset is a set of (input, verified-actual-output) pairs where the "actual output" comes from running real, trusted, deterministic code — not from guessing or hand-authoring — which is what makes it usable as objective ground truth.
Good Golden Dataset entries are deliberately designed, not randomly generated: isolate one rule at a time, include a fully-passing baseline, include boundary cases, and include cascading/multi-rule failures.
Anything time-dependent in a test case (like age) must be pinned to a fixed reference date inside the test data itself, or the "correct" answer silently drifts.
An agent can be evaluated against a Golden Dataset in two distinct modes: code comprehension, or business-rule application (useful even before code exists).
Scoring only the final yes/no verdict hides real, business-relevant errors — rule-level precision, recall, and cross-run consistency reveal problems that verdict-only accuracy completely misses.
The payoff is a scorecard, run repeatedly across every entry, that turns "does this agent seem good?" into a specific, trackable, comparable number.

↑ Back to Table of Contents

7. Glossary

Term	Meaning
Golden Dataset	A curated set of (input, verified ground-truth output) pairs used to score an agent's predictions against a trustworthy reference.
Ground truth	The actual, verified correct answer for a test case — here, produced by running deterministic code, not by guessing.
Boundary case	A test input crafted to sit exactly on a pass/fail threshold, designed to catch off-by-one-style logic errors.
Cascading failure	A single root cause (e.g., one missing field) that correctly trips more than one rule at once.
Rule-level precision	Of the failures an agent reported, the fraction that were real.
Rule-level recall	Of the real failures that existed, the fraction the agent actually found.
Cross-run agreement / self-consistency	How much an agent's own answers agree with each other across repeated runs on the identical input.

↑ Back to Table of Contents

This is Chapter 3 in the Eval Basics series. Chapter 1 explained why agents are inconsistent, Chapter 2 explained how to evaluate them without a single correct answer, and this chapter showed the concrete mechanics of building a Golden Dataset. Next: how do you turn a Golden Dataset and a scorecard like this into something a whole team trusts and watches over time — dashboards, thresholds, and what "good enough to ship" actually means for an AI agent. — continue to Chapter 4: Evals vs. Classic QA →

Table of Contents

1. The App We're Testing: Woody's Hardware Credit Application Validator

2. The 20 Business Rules

3. The Rule Engine — Code That Implements the Rules

4. Building the Golden Dataset

The synthetic JSON test cases

GD-001 — Fully eligible baseline

GD-002 — Underage applicant (fails R1 only)

GD-003 — Non-operating state (fails R5 only)

GD-004 — Bankruptcy in the last 7 years (fails R10 only)

GD-005 — High debt-to-income ratio (fails R8 only)

GD-006 — Missing credit score (cascading failure: R11 + R20)

GD-007 — Every threshold hit exactly on the boundary (should still be eligible)

GD-008 — Three simultaneous, unrelated failures (fails R4, R12, R13)

GD-009 — Credit score one point below minimum (fails R11 only)

GD-010 — Malformed SSN (fails R2 only)

5. Using the Golden Dataset as an Eval

Two ways to run this eval

What to score — not just yes/no

Worked example: GD-008, run twice

Scaling this to a real scorecard

6. Key Takeaways

7. Glossary