📰 Daily Briefing

Backpropagation — the chain rule, industrialised

Intuition. Yesterday's neuron computes a = σ(w·x + b); a network stacks thousands of them. Training needs the gradient of the loss with respect to every weight — the “credit assignment” problem: how much did this one weight, buried in layer 2, contribute to the error at the output? Backpropagation answers it exactly, for all weights at once, in a single backward sweep. It is not a new mathematical idea — it is the chain rule (covered 03 Jul) applied systematically, with intermediate results cached and reused.

The mechanism. The forward pass computes and stores each layer's pre-activation zˡ = Wˡaˡ⁻¹ + bˡ and activation aˡ = σ(zˡ). The backward pass then walks the layers in reverse, carrying one quantity per layer: the error signal δˡ = ∂L/∂zˡ. At the output it starts directly from the loss, e.g. for MSE: δᴰ = (aᴰ − y) ⊙ σ′(zᴰ). Each earlier layer receives its error from the layer above through the transposed weight matrix: δˡ = (Wˡ⁺¹)ᵀ δˡ⁺¹ ⊙ σ′(zˡ). Read that formula slowly — it says: take the blame arriving from above, route it backwards along the same connections the signal travelled forwards, then gate it by how responsive the neuron was (σ′).

From error signals to gradients. Once δˡ is known, the gradients are immediate: ∂L/∂Wˡ = δˡ (aˡ⁻¹)ᵀ and ∂L/∂bˡ = δˡ. Each weight's gradient is simply (error at its output neuron) × (activation at its input neuron) — a weight is blamed in proportion to how strongly it was transmitting when the mistake happened.

Why it matters. The naive alternative — nudge each weight, re-run the network, measure the loss change — costs one forward pass per parameter: a 7B-parameter model would need 7 billion forward passes for one update. Backprop delivers all 7 billion gradients for roughly the cost of two forward passes. That factor is the entire economic feasibility of deep learning. It also explains yesterday's vanishing-gradient remark: each backward step multiplies by σ′(z), and sigmoid's derivative maxes out at 0.25 — ten sigmoid layers shrink the signal by up to 0.25¹⁰ ≈ 10⁻⁶. ReLU's derivative of 1 is what lets error signals survive the trip through deep stacks.

import numpy as np
rng = np.random.default_rng(0)

# 2-2-1 network learning XOR, trained by hand-written backprop
X = np.array([[0,0],[0,1],[1,0],[1,1]], float)
y = np.array([[0],[1],[1],[0]], float)

W1, b1 = rng.normal(0,1,(2,2)), np.zeros(2)
W2, b2 = rng.normal(0,1,(2,1)), np.zeros(1)
sig  = lambda z: 1/(1+np.exp(-z))

for step in range(5000):
    # ---- forward (cache z1,a1,a2) ----
    z1 = X@W1 + b1;  a1 = np.maximum(0, z1)      # ReLU hidden
    a2 = sig(a1@W2 + b2)                          # sigmoid output
    # ---- backward ----
    d2 = (a2 - y)                                 # BCE+sigmoid shortcut
    d1 = (d2 @ W2.T) * (z1 > 0)                   # route blame, gate by ReLU'
    # ---- gradient step ----
    W2 -= 0.1 * a1.T@d2;  b2 -= 0.1 * d2.sum(0)
    W1 -= 0.1 * X.T@d1;   b1 -= 0.1 * d1.sum(0)

print(a2.round(2).ravel())   # -> [0.01 0.98 0.98 0.02]  XOR learned

Every deep-learning framework (PyTorch's loss.backward(), JAX's grad) is an automated, graph-general version of these six backward lines — reverse-mode automatic differentiation.