LESSON 2.3 · Theory · 100 min
Embeddings: place discrete symbols in space
Embeddings: place discrete symbols in space. Learn lookup tables, dense representations, similarity, and context windows and complete: Visualize character embeddings. Part of the “
Learning objectives
- Explain what problem “Embeddings: place discrete symbols in space” solves without hiding behind terminology.
- Trace the variables and causal links across lookup tables, dense representations, similarity.
- Complete “Visualize character embeddings” and judge the result with evidence rather than intuition.
Core concepts
lookup tables
lookup tables is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
dense representations
dense representations is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
similarity
similarity is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and context windows
and context windows is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Visualize character embeddings
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.