LESSON 5.6 · Engineering · 140 min
Evaluation: define “good” before optimizing
Evaluation: define “good” before optimizing. Learn perplexity, task evaluation, LLM judges, contamination, and variance and complete: Build a minimal evaluation harness. Part of th
Learning objectives
- Explain what problem “Evaluation: define “good” before optimizing” solves without hiding behind terminology.
- Trace the variables and causal links across perplexity, task evaluation, LLM judges.
- Complete “Build a minimal evaluation harness” and judge the result with evidence rather than intuition.
Core concepts
perplexity
perplexity is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
task evaluation
task evaluation is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
LLM judges
LLM judges is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
contamination
contamination is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and variance
and variance is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Build a minimal evaluation harness
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.