LESSON 4.2 · Engineering · 140 min
Data engineering: collect, clean, deduplicate, mix
Data engineering: collect, clean, deduplicate, mix. Learn quality filtering, MinHash, contamination, and mixtures and complete: Build a traceable 1 GB training corpus. Part of the
Learning objectives
- Explain what problem “Data engineering: collect, clean, deduplicate, mix” solves without hiding behind terminology.
- Trace the variables and causal links across quality filtering, MinHash, contamination.
- Complete “Build a traceable 1 GB training corpus” and judge the result with evidence rather than intuition.
Core concepts
quality filtering
quality filtering is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
MinHash
MinHash is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
contamination
contamination is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and mixtures
and mixtures is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Build a traceable 1 GB training corpus
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.