// topic 04
Evals
Offline evals before shipping, and online evals once a system is live — what to measure, how to build a test set, and how to trust the result.
Overview
Placeholder — why "it seemed to work when I tried it" doesn't scale, and the two eval regimes (pre-ship, in-production) this topic covers.
Offline Evals (pre-ship)
Placeholder — test-set construction, golden examples, grading methods (exact match, rubric, LLM-judge).
| Method | Good for |
|---|---|
| Exact / structural match | Deterministic outputs (JSON shape, tool calls) |
| Rubric scoring | Open-ended quality with defined criteria |
| LLM-as-judge | Scaling subjective review, with spot-checked calibration |
Online Evals (in production)
Placeholder — sampling real traffic, feedback signals, drift detection, alerting thresholds.
Open question: what's the minimum signal needed to catch a regression before a user escalates it?
Running notes
Placeholder — day-by-day notes go here.