📌 Daily Briefing

Batch Normalization — keeping every layer's inputs well-behaved

Intuition. As a deep network trains, the weights in early layers keep changing — which means the distribution of inputs arriving at later layers keeps shifting under their feet. Each layer is trying to learn on moving ground. Batch normalization (Ioffe & Szegedy, 2015) fixes this by re-standardizing the activations inside the network: for each feature, subtract the mini-batch mean and divide by the mini-batch standard deviation. Every layer then sees inputs that are roughly zero-mean, unit-variance — training becomes dramatically more stable.

The mechanism. For a mini-batch of activations x₁…x_m in one channel: compute μ = mean(x) and σ² = var(x), normalize x̂ᵢ = (xᵢ − μ) / √(σ² + ε), then rescale with two learnable parameters: yᵢ = γ·x̂ᵢ + β. The γ and β are crucial — they let the network undo the normalization if that's actually better, so BatchNorm never reduces what the layer can express. Modern analysis suggests its real benefit is less "fixing covariate shift" and more that it smooths the loss landscape, allowing much larger learning rates without divergence.

Why it matters. BatchNorm made very deep networks practical: 2–10× faster training, tolerance of high learning rates, less sensitivity to initialization, and a mild regularization effect (each example's output depends on which batch-mates it was normalized with — a source of noise, like a soft dropout). One subtlety you must know for deployment: at inference there is no batch, so the layer switches to running averages of μ and σ² collected during training — this train/eval difference is the classic source of the "model works in training, breaks in production" bug (model.eval() in PyTorch). Its cousin LayerNorm — which normalizes across features instead of across the batch — is what Transformers use, since language batches have variable lengths.

import numpy as np

def batchnorm_forward(x, gamma, beta, eps=1e-5):
    mu, var = x.mean(axis=0), x.var(axis=0)   # per-feature stats over the batch
    x_hat = (x - mu) / np.sqrt(var + eps)     # normalize: mean 0, var 1
    return gamma * x_hat + beta               # learnable rescale + shift

x = np.random.randn(32, 4) * 7 + 3           # messy activations (batch of 32)
y = batchnorm_forward(x, gamma=np.ones(4), beta=np.zeros(4))
print(y.mean(axis=0).round(3), y.std(axis=0).round(3))  # ≈ [0 0 0 0] [1 1 1 1]