LESSON 5.7 · Diagnosis · 100 min
Hallucination, calibration, and uncertainty
Hallucination, calibration, and uncertainty. Learn factuality, abstention, confidence, and RAG boundaries and complete: Design a falsifiable hallucination benchmark. Part of the “M
Learning objectives
- Explain what problem “Hallucination, calibration, and uncertainty” solves without hiding behind terminology.
- Trace the variables and causal links across factuality, abstention, confidence.
- Complete “Design a falsifiable hallucination benchmark” and judge the result with evidence rather than intuition.
Core concepts
factuality
factuality is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
abstention
abstention is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
confidence
confidence is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and RAG boundaries
and RAG boundaries is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Design a falsifiable hallucination benchmark
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.