📌 Daily Briefing

Stochastic Gradient Descent & mini-batches — training on data you can't fit in memory

Intuition. Gradient descent (Wed's lesson) computes the loss gradient over the entire dataset before taking one step. With millions of examples that's absurdly expensive — and wasteful, because 10,000 random examples already tell you almost the same direction as 10 million. SGD embraces this: estimate the gradient from a small random mini-batch, step immediately, repeat.

The mechanism. Instead of the full-batch update, we sample a batch B (say 32–512 examples) and update with w ← w − η · ∇LB(w), where LB is the average loss over the batch only. Because batches are sampled randomly, ∇LB is an unbiased estimator of the true gradient: correct on average, noisy per step. One pass over all batches is an epoch; data is reshuffled each epoch so batches differ.

Why the noise is a feature, not a bug. The jitter in each step lets the optimizer bounce out of narrow, sharp minima and shallow saddle points that would trap exact gradient descent — empirically, SGD's noise acts like a regularizer and finds flatter minima that generalize better. Batch size is the knob: small batches = noisier, more regularizing, less hardware-efficient; large batches = smoother, faster on GPUs, but may need learning-rate tuning to match generalization.

rng = np.random.default_rng(0)
for epoch in range(epochs):
    idx = rng.permutation(len(X))          # reshuffle each epoch
    for b in range(0, len(X), 64):         # batch_size = 64
        i = idx[b:b+64]
        grad = X[i].T @ (X[i] @ w - y[i]) / len(i)   # ∇MSE on batch
        w -= lr * grad                     # step immediately

Why it matters. Every modern model — from logistic regression to trillion-parameter LLMs — is trained with mini-batch SGD or a descendant of it (momentum, Adam — coming up next). Batch size, learning rate and epochs are the first three hyperparameters you will ever tune.

Learning progress

Phase 5 of 20 · 15 concepts covered so far