LESSON wm.0.2 · Derivation · 110 min
POMDPs: modeling a world you cannot fully observe
POMDPs: modeling a world you cannot fully observe. Learn latent state, belief state, observation model, reward, and policy and complete: Hand-calculate belief updates in a partiall
Learning objectives
- Explain what problem “POMDPs: modeling a world you cannot fully observe” solves without hiding behind terminology.
- Trace the variables and causal links across latent state, belief state, observation model.
- Complete “Hand-calculate belief updates in a partially observable maze” and judge the result with evidence rather than intuition.
Core concepts
latent state
latent state belongs to a partially observable decision process: hidden state produces observations, action changes future state, and the agent updates a belief from its action-observation history before comparing expected returns. Keep ground-truth state, visible observation, and estimated belief separate in the experiment.
belief state
belief state belongs to a partially observable decision process: hidden state produces observations, action changes future state, and the agent updates a belief from its action-observation history before comparing expected returns. Keep ground-truth state, visible observation, and estimated belief separate in the experiment.
observation model
observation model belongs to a partially observable decision process: hidden state produces observations, action changes future state, and the agent updates a belief from its action-observation history before comparing expected returns. Keep ground-truth state, visible observation, and estimated belief separate in the experiment.
reward
reward belongs to a partially observable decision process: hidden state produces observations, action changes future state, and the agent updates a belief from its action-observation history before comparing expected returns. Keep ground-truth state, visible observation, and estimated belief separate in the experiment.
and policy
and policy belongs to a partially observable decision process: hidden state produces observations, action changes future state, and the agent updates a belief from its action-observation history before comparing expected returns. Keep ground-truth state, visible observation, and estimated belief separate in the experiment.
Build and verify
Hand-calculate belief updates in a partially observable maze
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.