Budget before training: parameters, FLOPs, memory

DIRECT ANSWER · VERIFIED SOURCES · 2026-07-22

How should you estimate parameter count, memory, and compute before training an LLM?

Build three separate budgets: sum matrix shapes for parameter count; account independently for weights, gradients, optimizer state, and activations for training memory; and sum operator FLOPs for compute. For a standard dense Transformer, C ≈ 6ND is a useful order-of-magnitude training estimate, where N is parameter count and D is training tokens, but the final plan should use operator-level estimates and measured peaks for the target configuration.

Lecture alignment

Course alignment: hand-count a small model’s parameter matrices, separately record parameter/gradient/optimizer/activation memory, then compare the estimate with a profiler instead of memorizing one bytes-per-parameter rule.

Key takeaways

• Parameter-state memory depends on precision and optimizer; mixed precision may retain master weights, so no universal bytes-per-parameter number applies.
• Activation memory grows with batch, sequence length, layers, and width; activation checkpointing trades recomputation for memory.
• FLOPs estimate total work, while hardware utilization determines elapsed time; communication, bandwidth, and data loading can also bottleneck.

Boundary and caveat

The 6ND approximation omits some sequence-length attention cost, vocabulary projection, sparse/MoE behavior, recomputation, and communication. Always report dtype, optimizer, parallelism, sequence length, and checkpointing assumptions with a budget.

Primary sources

• Stanford CS336 · Language Modeling from Scratch
• Hoffmann et al. · Training Compute-Optimal LLMs
• PyTorch · Activation Checkpointing

parameter counts

parameter counts is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.

activation memory

activation memory is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

optimizer states

optimizer states is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

and MFU

and MFU is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.