LESSON 0.4 · Theory · 110 min
Probability, information, and cross-entropy
Probability, information, and cross-entropy. Learn conditional probability, expectation, entropy, KL divergence, and maximum likelihood and complete: Derive and calculate NLL from
Learning objectives
- Explain what problem “Probability, information, and cross-entropy” solves without hiding behind terminology.
- Trace the variables and causal links across conditional probability, expectation, entropy.
- Complete “Derive and calculate NLL from a count distribution” and judge the result with evidence rather than intuition.
Core concepts
conditional probability
conditional probability is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
expectation
expectation is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
entropy
entropy is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
KL divergence
KL divergence is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and maximum likelihood
and maximum likelihood is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Derive and calculate NLL from a count distribution
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.