📌 Daily Briefing

Gradient descent — the algorithm that trains everything

Intuition. You are standing on a foggy mountainside and want to reach the valley. You can't see the valley, but you can feel the slope under your feet. Strategy: take a small step in the steepest downhill direction, feel the slope again, repeat. That is gradient descent. The mountain is the loss function, your position is the model's parameters, and the valley floor is the parameter setting with the smallest error. Yesterday's linear regression secretly used it; today we look at the algorithm itself, because this exact loop — unchanged — is what trains a 2-trillion-parameter LLM.

The mechanism. The gradient ∇L(w) points in the direction of steepest increase of the loss, so we step against it: w ← w − η·∇L(w). The learning rate η controls the step size, and it is the single most important hyperparameter in deep learning. Too small: training crawls, thousands of tiny steps. Too large: you overshoot the valley and bounce between the walls — the loss oscillates or explodes. Just right: fast, stable descent. For a bowl-shaped (convex) loss like MSE, gradient descent with a suitable η provably reaches the global minimum; for neural networks the landscape has many valleys, yet in practice good ones are almost always found.

Batches. Computing ∇L over the entire dataset per step ("batch" gradient descent) is exact but slow. Stochastic gradient descent (SGD) estimates the gradient from one example — noisy but cheap. The practical sweet spot is the mini-batch: average the gradient over, say, 32–1024 examples. The noise even helps, jiggling the optimizer out of bad shallow valleys. This is why GPUs and batch size matter so much in AI engineering.

import numpy as np
# minimize L(w) = (w-3)^2  — minimum obviously at w = 3
grad = lambda w: 2*(w - 3)
for eta in [0.01, 0.3, 1.05]:
    w = 0.0
    for _ in range(50):
        w -= eta * grad(w)
    print(f"eta={eta:<5} → w = {w:.4f}")
# eta=0.01  → w = 1.9292   (too small: still far away)
# eta=0.3   → w = 3.0000   (just right)
# eta=1.05  → w = -358.83  (too large: diverged!)

Why it matters. Every "learning" you will meet from here on — backpropagation, Adam, RLHF fine-tuning — is gradient descent plus refinements. Backprop (phase 4) is just an efficient way to compute ∇L; Adam (phase 5) is gradient descent with per-parameter adaptive step sizes. Internalize the update rule and the learning-rate trade-off now, and the rest of the curriculum becomes variations on a theme.

Curriculum: AI Engineering from Scratch