📌 Daily Tech · AI · Jobs Briefing

Random Variables, Expectation & Variance — the units of uncertainty

Intuition. A random variable is just a number whose value depends on chance: the roll of a die, the next token a language model picks, the error your model makes on a random test example. Instead of tracking every possible outcome, we summarize the variable with two numbers: its expectation (the long-run average) and its variance (how far outcomes typically spread around that average).

The math. For a discrete random variable X taking values x with probabilities p(x), the expectation is E[X] = Σ x·p(x) — a probability-weighted average. Variance is Var(X) = E[(X − E[X])²] = E[X²] − (E[X])², the average squared distance from the mean. Its square root, the standard deviation, is in the same units as X. Expectation is linear — E[aX + bY] = aE[X] + bE[Y] — even when X and Y are dependent, which makes it the single most-used algebraic fact in ML derivations.

Why it matters for AI. Nearly every training objective is an expectation: the loss you minimize is E[ℓ(model(x), y)] over the data distribution. You can never compute it exactly — you estimate it with a mini-batch average, which is itself a random variable whose variance shrinks as 1/n with batch size n. That is why bigger batches give smoother gradients, why SGD is noisy, and why variance-reduction tricks (baselines in RLHF, control variates) exist. Sampling temperature in LLMs is variance control too: raising it spreads probability mass, increasing the variance of the chosen tokens.

import numpy as np
rng = np.random.default_rng(0)

# X = die roll: E[X] = 3.5, Var(X) = 35/12 ≈ 2.92
rolls = rng.integers(1, 7, size=100_000)
print(rolls.mean(), rolls.var())        # ≈ 3.5, ≈ 2.92

# Mini-batch means: variance shrinks as 1/n
for n in (1, 10, 100):
    means = rolls[:10_000*n].reshape(-1, n).mean(axis=1)
    print(n, means.var())               # ≈ 2.92/n

Takeaway. Expectation tells you what to optimize; variance tells you how trustworthy your estimate of it is. Most of practical deep learning — batch sizes, learning-rate schedules, evaluation error bars — is a negotiation between the two. Source inspiration: aiengineeringfromscratch.com