Table of Contents
Chapter 1 — How LLMs Work, and Why They're Not Consistent
1According to Chapter 1, what is fundamentally happening when an LLM generates a response?
2Which setting most directly controls how much randomness is applied when an LLM selects its next token?
3Besides sampling randomness, Chapter 1 names another cause of inconsistency that has nothing to do with the prompt at all. What is it?
Chapter 2 — What Is an Agent, and How Do You Judge a Good One?
4What is the key architectural difference between a plain LLM and an agent, according to Chapter 2?
5Chapter 2 argues the biggest difference between a "smart" agent and a "not-so-smart" agent is:
6Which of the following is NOT one of the eval methods described in Chapter 2 for judging agent output without a single correct answer?
Chapter 3 — Building a Golden Dataset
7In the Woody's Hardware Golden Dataset, what makes the "expected output" for each test case trustworthy ground truth?
8Why does the applicationDate field matter in the Golden Dataset design?
9What does GD-007 (the boundary case) specifically test?
Chapter 4 — Evals vs. Classic QA
10What foundational assumption does classic QA (unit, integration, UAT, regression testing) rely on that doesn't hold for LLM-based agents?
11According to Chapter 4, how does the mindset around a "flaky" result differ between classic QA and evals?
12In Chapter 4's ship-gate framework, what is the difference between a "hard gate" and a "soft threshold"?
Chapter 5 — Goodhart's Law
13How is Goodhart's Law usually phrased, as stated in Chapter 5?
14In Chapter 5's worked example, the agent that always flags all 20 rules as failing achieved 100% recall and 100% self-consistency. What did its precision score reveal?
15What is the key difference between "overfitting to a Golden Dataset" and "Goodhart's Law," as Chapter 5 distinguishes them?
16Which practice does Chapter 5 recommend specifically to keep a benchmark honest as agents keep improving?
Chapter 6 — The Ten Agent Fault Lines, and How Many Runs Are Enough
17Which three clusters organize the Ten Agent Fault Lines in Chapter 6?
18"Misaligned Metrics" — an agent optimizing for speed/volume over quality — is explicitly named in Chapter 6 as a real-world instance of which concept from an earlier chapter?
19According to Chapter 6's statistics section, if you want to cut your margin of error in half, how many times more eval runs do you generally need?
20Chapter 6 notes that agents with a true pass rate near which value are statistically the hardest to measure precisely with a small number of runs?
Chapter 7 — Applying LangSmith
21In Chapter 7's mapping, which LangSmith capability most directly addresses Chapter 6's "Black Box Behavior" fault line by giving visibility into an agent's reasoning?
22In Chapter 7's worked ROI example (100 features, ~1,000,000 lines of code, 20 engineers), what happened to the "manual review tax" once an eval practice was in place?
23Why does Chapter 7 argue that "higher quality" and "shorter delivery time" are not a tradeoff in the ROI example, but can improve together?
Chapter 8 — The LangSmith Training Plan
24Which of the following is listed in Chapter 8 as a core technical prerequisite skill (as opposed to an agentic-AI-specific skill) for learning LangSmith effectively?
25According to Chapter 8's AI SDLC table, which fault lines are called out as most likely to affect a code-review agent specifically?
Scoring Guide
This chapter is a checkpoint, not a new concept — every question traces back to something already taught in Chapters 1 through 8. Missing a question is a pointer to exactly which chapter is worth a second pass, not a sign to start over. Use your live score above once you've answered all 25 to see where you land.
| Score | What it suggests |
|---|---|
| 22–25 correct | Solid grasp of the full series — ready to move into hands-on practice (Chapter 8's training plan). |
| 16–21 correct | Good foundation with a few gaps — revisit the specific chapters behind any missed questions before starting hands-on work. |
| Below 16 correct | Worth a full re-read of Chapters 1–8 in order before attempting the training plan — the chapters build on each other deliberately. |
This chapter is a checkpoint, not a new concept — every question traces back to something already taught in Chapters 1 through 8. — Back to All Chapters →