LESSON 8.7 · Code Study · 180 min

Kimi K3: audit a 3T-class open-weight system

Kimi K3: audit a 3T-class open-weight system. Learn KDA, AttnRes, Stable LatentMoE, MXFP4 QAT, million-token agentic RL, and the Kimi K3 License and complete: Build a five-layer au

Learning objectives

  1. Draw the sequence-, depth-, and width-wise information paths behind KDA, AttnRes, and Stable LatentMoE.
  2. Separate 2.8T total parameters from 104B active parameters, then account for storage, communication, KV state, and runtime overhead.
  3. Audit an evaluation claim by matching reasoning effort, agent harness, tool access, benchmark split, cost, and preserved failures.
  4. Distinguish open weights from unrestricted open source and identify when the Kimi K3 License requires a separate commercial review.

Core concepts

KDA

Kimi Delta Attention supplies linear recurrent attention for most layers, while periodic Gated MLA layers restore global content-based access. Audit the 3:1 pattern and the final global-attention layer rather than reducing the design to “linear attention”.

AttnRes

Attention Residuals let a block selectively read earlier block outputs and embeddings. Track which depth-wise paths are available and what extra state or communication they require.

Stable LatentMoE

The model routes through 896 experts and selects 16 per token in a compact latent expert space. RMSNorm, SiTU-GLU, and quantile balancing are stabilization mechanisms, not optional footnotes.

MXFP4 QAT

Quantization-aware training begins during post-training and targets MXFP4 expert weights with MXFP8 activations. Do not infer that the entire checkpoint or serving stack occupies four bits per parameter.

Million-token agentic RL

The post-training stack combines SFT, long-horizon agentic RL, preserved thinking history, and multiple effort levels. A one-million-token window is capacity; usable long-horizon behavior still needs trajectory and compaction tests.

Build and verify

Build a five-layer audit matrix spanning architecture, training, inference, evaluation, and licensing

  • Predict: write what 2.8T total, 104B active, 896 routed, 16 selected, and a one-million-token context do—and do not—imply.
  • Build: give every claim a source, evidence class, measurement boundary, dependency, and falsification test.
  • Verify: cross-check the technical report, official repository, and exact license text; label official claims separately from independent evidence.
  • Transfer: apply the same matrix to another open-weight model and record which fields cannot be compared directly.

Open the complete interactive lesson