☀️ Daily Briefing

Intuition. Yesterday's SGD update follows the current mini-batch gradient — and only that. But mini-batch gradients are noisy, and loss surfaces have narrow ravines: steep walls in one direction, a gentle slope in another. Plain SGD zig-zags across the ravine walls instead of rolling down the valley floor. Momentum fixes this like a heavy ball: keep a running velocity, add a fraction of the previous step to the new one. Oscillations cancel out; the consistent downhill direction accumulates.

The mechanism. Momentum keeps an exponential moving average of gradients: v ← β·v + g, then w ← w − lr·v (β ≈ 0.9, so roughly the last ~10 gradients vote on the direction). Adam goes one step further and adapts the learning rate per parameter. It tracks two EMAs — the mean of gradients m (first moment) and of squared gradients v (second moment) — and divides one by the square root of the other:

m = β1·m + (1−β1)·g        # direction (momentum)
v = β2·v + (1−β2)·g²       # magnitude (per-weight scale)
m̂ = m/(1−β1ᵗ);  v̂ = v/(1−β2ᵗ)   # bias correction
w = w − lr · m̂ / (√v̂ + ε)

Parameters with rare, small gradients (e.g. embeddings of rare tokens) get boosted; parameters with huge, frequent gradients get damped. The bias correction matters early in training, when the EMAs are still warming up from zero.

Why it matters. Adam (2014) is the default optimizer of the deep-learning era — nearly every Transformer, including the frontier LLMs, trains with AdamW (Adam with decoupled weight decay). Its per-parameter scaling makes training robust to the wildly different gradient magnitudes across hundreds of layers, and it usually works with minimal learning-rate tuning: lr=3e-4 is a famous "just works" starting point.

# Tiny Adam in NumPy
m = v = 0; b1, b2, lr, eps = .9, .999, 3e-4, 1e-8
for t in range(1, steps+1):
    g = grad(w)
    m = b1*m + (1-b1)*g
    v = b2*v + (1-b2)*g**2
    mhat = m/(1-b1**t); vhat = v/(1-b2**t)
    w -= lr * mhat/(np.sqrt(vhat)+eps)

Rule of thumb: SGD+momentum can generalize slightly better on vision tasks; Adam/AdamW wins on Transformers and anything sparse. Next up in this phase: regularization — how to stop all this optimization power from memorizing the training set.

Learning progress

Phase 5 of 20 · 16 concepts covered