C27
LLM Evaluation and LLM-as-a-Judge
7 lessons3.5 hoursPlannedFree
Outcome
Design evaluations for LLM systems and build LLM judges you can trust, validated against human labels.
Prerequisites
- Recommended: C2 Statistics for Practitioners
- Recommended: C17 RAG End to End
Lessons
- L01Designing an eval: tasks, datasets, and what 'good' means · coming soonExercise: Write an eval spec for a customer-support bot.
- L02Reference-based and programmatic checks · coming soonExercise: Score outputs with exact match, schema validation, and unit tests.
- L03LLM-as-a-judge: rubrics, single scores, and pairwise comparison · coming soonExercise: Build a rubric judge and a pairwise judge for the same task.
- L04Judge biases: position, verbosity, self-preference · coming soonExercise: Measure position bias by swapping answer order.
- L05Validating a judge against human labels · coming soonExercise: Compute Cohen's kappa between judge and human labels on 100 items.
- L06Statistics for evals: intervals and paired comparisons · coming soonExercise: Decide whether model B beats model A with a paired test and a confidence interval.
- L07Running evals in CI and tracking them over time · coming soonExercise: Fail a build when a key metric regresses beyond its noise band.