LESSON 6.4 · Engineering · 130 min
llama.cpp: dissect local inference engineering
llama.cpp: dissect local inference engineering. Learn GGUF, mmap, CPU/GPU offload, and sampling and complete: Deploy and benchmark on a Mac. Part of the “Make Models Respond Fast a
Learning objectives
- Explain what problem “llama.cpp: dissect local inference engineering” solves without hiding behind terminology.
- Trace the variables and causal links across GGUF, mmap, CPU/GPU offload.
- Complete “Deploy and benchmark on a Mac” and judge the result with evidence rather than intuition.
Core concepts
GGUF
GGUF is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
mmap
mmap is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
CPU/GPU offload
CPU/GPU offload is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and sampling
and sampling is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Deploy and benchmark on a Mac
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.