📌 Daily Tech · AI · Learning Briefing

AI Concept

The Hessian Matrix — second-order curvature

Intuition. If the gradient tells you the slope of a loss surface — which way is downhill and how steep — the Hessian tells you the curvature: how that slope itself is changing. It's the multivariable analogue of the second derivative. A large positive curvature means a sharp, narrow valley; near-zero curvature means a flat plateau. This is exactly the information first-order gradient descent ignores.

The mechanism / math. For a scalar function f(x) with input vector x ∈ ℝⁿ, the Hessian H is the n×n matrix of all second partial derivatives:

By Schwarz's theorem, if f is nicely smooth then Hij = Hji, so H is symmetric. Its eigenvalues describe curvature along principal directions: all positive → a local minimum (bowl), all negative → a local maximum, mixed signs → a saddle point. The second-order Taylor expansion around a point makes this concrete:

Why it matters for AI. Newton's method uses H⁻¹ to jump straight toward the minimum of that quadratic approximation, converging far faster than plain gradient descent near a minimum. In deep learning the full Hessian is usually too big to store (n = millions of parameters → n² entries), so we use approximations — L-BFGS, Gauss-Newton, K-FAC, or cheap Hessian-vector products. The Hessian's eigenvalue spectrum also explains why training is hard: a large condition number (ratio of largest to smallest eigenvalue) means steep in some directions and flat in others, which is why we need momentum, Adam, and learning-rate tuning. Saddle points — where H has mixed-sign eigenvalues — are now understood to be the main obstacle in high-dimensional loss landscapes, not local minima.

Tiny worked example. For f(x,y) = x² + 3xy + y²:

∂f/∂x = 2x + 3y      ∂f/∂y = 3x + 2y

H = [ ∂²f/∂x²   ∂²f/∂x∂y ] = [ 2  3 ]
    [ ∂²f/∂y∂x  ∂²f/∂y²  ]   [ 3  2 ]

eigenvalues(H) = 5 and -1  →  mixed signs  →  saddle point

# NumPy check
import numpy as np
H = np.array([[2., 3.],[3., 2.]])
print(np.linalg.eigvalsh(H))   # [-1.  5.]  → saddle

AI Engineering from Scratch →