Daily Tech · AI · Deutsch · Steuer

The Chain Rule — how gradients flow through composed functions

Intuition. Neural networks are just deeply nested functions: an output is a function of a hidden layer, which is a function of another layer, which is a function of the input. To train them we need the derivative of the final loss with respect to every weight buried deep inside. The chain rule is the single tool that lets us do this: it says the sensitivity of an output to a distant input is the product of the local sensitivities along the path connecting them. Think of gears meshing — turn the first, and the effect multiplies down the chain.

The mechanism. If y = f(u) and u = g(x), then

dy/dx = (dy/du) · (du/dx)

Each factor is a local derivative, evaluated at the point you're currently at. For a longer composition y = f(g(h(x))) you simply multiply more links: dy/dx = f'·g'·h'. When variables branch and recombine (as in real networks), you sum the contributions over all paths — this is the multivariable chain rule.

Why it matters. Backpropagation is nothing more than the chain rule applied efficiently across a computation graph. During the forward pass we store each intermediate value; during the backward pass we walk the graph in reverse, multiplying local derivatives and reusing shared sub-results instead of recomputing them. That reuse is what makes training networks with billions of parameters tractable — the cost of the backward pass is roughly the same as the forward pass, not exponentially larger.

Worked example. Take the loss L = (σ(w·x) − t)² with sigmoid σ. Let z = w·x, a = σ(z), L = (a − t)². Chain the three links:

dL/dw = dL/da · da/dz · dz/dw
      = 2(a − t) · a(1 − a) · x

Read right to left and that is exactly the gradient backprop computes for a single sigmoid neuron. A tiny numeric check:

import numpy as np
x, w, t = 2.0, 0.5, 0.0
z = w * x                 # 1.0
a = 1/(1+np.exp(-z))      # 0.7311
dL_dw = 2*(a-t) * a*(1-a) * x   # -> 0.5749
print(dL_dw)

Nudging w by -η·dL/dw lowers the loss — one gradient-descent step, powered entirely by the chain rule. Tomorrow this leads naturally into the perceptron and full backprop.