A Training Plan for Learning LangSmith

0Prerequisites Check & Setup

• Confirm the Section 1 skill bars are genuinely met — don't skip this; it's the single biggest predictor of whether the rest of the plan lands.
• Read (or re-read) Chapters 1–6 of this series.
• Create a LangSmith account, install the SDK, generate an API key.
• Checkpoint: A working local environment that can successfully make one authenticated call to the LangSmith API.

1Tracing & Observability

Objective: understand what a trace actually captures, and instrument a real function with it.

from langsmith import traceable

@traceable(name="validate_credit_application")
def validate_application_with_agent(application: dict) -> dict:
    # calls the coding agent / LLM to analyze the Woody's Hardware application
    # (see Chapter 3 for the full 20-rule business logic being analyzed)
    ...
    return agent_result

• Wrap a function that asks an agent to analyze a Woody's Hardware application (Mode A or Mode B from Chapter 3) with @traceable, run it, and inspect the resulting trace in the LangSmith UI.
• Identify, inside a single trace: the model call, any tool calls, latency, and token usage — mapping directly onto Chapter 2's think → act → observe loop, now visualized instead of theoretical.
• Checkpoint: Can walk a colleague through a real trace end to end, in plain language.

2Building a Golden Dataset in LangSmith

Objective: turn Chapter 3's ten hand-authored golden cases into an actual, reusable LangSmith dataset.

from langsmith import Client

client = Client()
dataset = client.create_dataset(
    dataset_name="woodys-hardware-golden-v1",
    description="10 golden credit application test cases (Chapter 3)",
)

for case in GOLDEN_CASES:  # GD-001 through GD-010
    client.create_example(
        inputs={"application": case["input"]},
        outputs={"expected": case["expected"]},
        dataset_id=dataset.id,
    )

• Load all ten GD-001 through GD-010 cases as dataset examples, preserving each case's documented rationale as metadata.
• Checkpoint: A LangSmith dataset with 10 examples, each traceable back to the specific Chapter 3 scenario it represents.

3Evaluators — From Chapter 2's Methods to Configured Pipelines

Objective: implement the eval methods from Chapter 2 as real, runnable LangSmith evaluators.

from langsmith.evaluation import evaluate

def rule_level_recall_evaluator(run, example):
    predicted_failed = set(run.outputs.get("failedRules", []))
    actual_failed = set(example.outputs["expected"]["failedRules"])
    if not actual_failed:
        return {"key": "recall", "score": 1.0}
    recall = len(predicted_failed & actual_failed) / len(actual_failed)
    return {"key": "recall", "score": recall}

results = evaluate(
    validate_application_with_agent,
    data="woodys-hardware-golden-v1",
    evaluators=[rule_level_recall_evaluator],
    experiment_prefix="agent-v1",
)

• Implement rule-level recall and precision evaluators (Chapter 3's metrics), an execution-based check, and one LLM-as-judge evaluator scoring explanation clarity.
• Run the full suite against the dataset from Phase 2 and review the per-case results.
• Checkpoint: Can run an eval suite on demand and correctly interpret a precision/recall breakdown, not just a single pass/fail number.

4Consistency, Statistics, and Ship Gates

Objective: apply Chapter 6's statistical rigor and Chapter 4's dashboard discipline inside LangSmith.

• Run the same golden case 5, then 30 times; compute the margin of error at each sample size and compare it against Chapter 6's worked table.
• Recreate Chapter 4's two-tier ship-gate table (hard gates on compliance-sensitive rules, soft thresholds elsewhere) as an actual dashboard view or saved query.
• Checkpoint: A documented ship-gate policy, with each threshold justified by a sample size the learner can defend, not picked arbitrarily.

5Prompt Iteration & Regression Safety

Objective: use the prompt playground and versioning to change an agent's behavior without shipping a silent regression.

• Deliberately modify the prompt used in Phase 1 (e.g., change how the 20 business rules are presented to the agent).
• Rerun the Phase 3 eval suite against both the old and new prompt versions, and compare them side by side on the dashboard.
• Practice identifying a regression and rolling back to the prior prompt version.
• Checkpoint: A documented before/after comparison of two prompt versions scored against the same Golden Dataset.

6Applying the Ten Agent Fault Lines

Objective: deliberately build eval coverage for the fault lines most relevant to the learner's actual coding-agent use case, using the SDLC-stage table from Section 2 to prioritize.

• Pick 3–4 fault lines from Chapter 6 that match the learner's real SDLC stage (e.g., a code-review agent should prioritize Hallucination and Black Box Behavior).
• For each chosen fault line, build one small dataset and evaluator pair using the LangSmith mechanism named in Chapter 7's mapping table.
• Checkpoint: A short internal write-up mapping each chosen fault line to the eval coverage actually built for it.

7Closing the Production Loop & Building the ROI Case

Objective: wire the whole system into something that runs continuously and can be defended to stakeholders.

• Configure a rule (or manual process) for promoting flagged real traces into the Golden Dataset — recreating Chapter 7's compounding loop.
• Integrate the Phase 3 eval suite into a CI pipeline, reusing the JUnit/Pytest integration patterns from Chapter 4.
• Populate Chapter 7's ROI model with the learner's own team's real or realistically estimated numbers, and draft a one-page summary.
• Checkpoint: A running, CI-integrated eval pipeline, plus a stakeholder-ready ROI one-pager.