LESSON 7.5 · Engineering · 130 min
Agent evaluation: from final answers to trajectories
Agent evaluation: from final answers to trajectories. Learn task success, trajectories, tool errors, cost, and latency and complete: Build a 30-task regression suite. Part of the “
Learning objectives
- Explain what problem “Agent evaluation: from final answers to trajectories” solves without hiding behind terminology.
- Trace the variables and causal links across task success, trajectories, tool errors.
- Complete “Build a 30-task regression suite” and judge the result with evidence rather than intuition.
Core concepts
task success
task success is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
trajectories
trajectories is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
tool errors
tool errors is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
cost
cost is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and latency
and latency is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Build a 30-task regression suite
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.