Daily Tech · AI · Deutsch · Steuer

The Jacobian — the derivative of a vector-valued function

Intuition. A gradient answers "how does one scalar output change as I nudge each input?" But most layers in a network map a whole vector of inputs to a whole vector of outputs. The Jacobian is the natural generalisation: it collects every partial derivative of every output with respect to every input into a single matrix. It is the best linear approximation of a vector-valued function at a point — the multi-dimensional version of the slope.

The mechanism. For a function f: ℝⁿ → ℝᵐ that maps x = (x₁,…,xₙ) to (f₁,…,fₘ), the Jacobian J is the m×n matrix

J[i][j] = ∂fᵢ / ∂xⱼ

Row i is exactly the gradient of the i-th output. So a gradient is just a Jacobian with one row (m = 1). Near a point x₀, the function behaves like f(x) ≈ f(x₀) + J·(x − x₀) — matrix–vector multiplication does the local linear map.

Why it matters. The chain rule for vectors becomes multiplication of Jacobians: if z = g(f(x)), then J_z = J_g · J_f. Backpropagation is precisely this — each layer has a Jacobian, and the gradient of the loss is propagated backward by multiplying by each layer's Jacobian transpose. Frameworks never materialise these huge matrices; instead they compute Jacobian–vector products (forward mode) and vector–Jacobian products (reverse mode / autograd), which give the same result far more cheaply.

Worked example. Let f(x₁,x₂) = (x₁²x₂,  sin x₁ + x₂). Differentiate each output w.r.t. each input:

J = [[ 2·x₁·x₂ ,  x₁²      ],
     [ cos x₁  ,  1        ]]

At (x₁,x₂) = (1, 3) this is [[6, 1], [cos 1 ≈ 0.540, 1]]. A quick check with autograd:

import torch
x = torch.tensor([1.0, 3.0], requires_grad=True)
f = lambda v: torch.stack([v[0]**2*v[1], torch.sin(v[0])+v[1]])
J = torch.autograd.functional.jacobian(f, x)
print(J)   # tensor([[6.0000, 1.0000],
           #         [0.5403, 1.0000]])

The autograd result matches the hand-derived matrix exactly. This vector-of-derivatives view is the bridge from single-variable calculus to how gradients actually flow through real network layers — next up, we start turning this machinery into a learning algorithm.