Mixed Precision

1. Precision is a resource

FP32 uses 32 bits per floating-point value. FP16 and bfloat16 use 16 bits, reducing memory traffic and often enabling much faster matrix multiplication on modern accelerators.

But lower precision represents a smaller or coarser numerical range.

2. FP16 and bfloat16 differ

FP16 offers more mantissa precision than bfloat16 but a much smaller exponent range. bfloat16 keeps an exponent range similar to FP32, making overflow/underflow easier to manage in many training workloads.

3. Mixed precision keeps critical work safer

Rather than convert everything blindly, frameworks use lower precision for many compute-heavy operations while keeping selected accumulations, optimizer state or master weights at safer precision.

4. Loss scaling

With FP16, very small gradients may underflow to zero. Multiply the loss by scale $S$ before backward:

$L'=SL.$

Gradients become $S$ times larger and are later unscaled before the optimizer update. Dynamic loss scaling adjusts $S$ when overflow is detected.

5. PyTorch idea

with torch.autocast(device_type='cuda'):
    pred = model(x)
    loss = criterion(pred, y)

Modern APIs can manage casting and scaling, but you should still monitor NaNs, infs and numerical equivalence.

6. Shape Check remains unchanged

Precision changes dtype, not tensor geometry:

[B,T,D] float32 → [B,T,D] bfloat16

What to Remember

Mixed precision is a systems/number-representation technique: spend high precision where it matters and low precision where hardware gains are large.

Next → Scaling Laws for Training: after understanding the mechanics of stable training, what predictable patterns appear when we scale model size, data and compute?