Daily Tech · AI · Deutsch · Steuer

Probability distributions & expectation — the language of uncertainty

Intuition. Phase 1 gave us the calculus machinery; phase 2 gives us the worldview. A modern model never says "the next word is cat" — it says "cat with probability 0.62, dog with 0.21, …". A probability distribution is just an assignment of belief across all possible outcomes, with two rules: every probability is ≥ 0, and they sum (or integrate) to 1. Everything an LLM outputs is a distribution over its vocabulary.

The mechanism. A random variable X takes values with probabilities p(x). For discrete outcomes, the distribution is a table (a categorical distribution — exactly what softmax produces). The single most useful summary is the expectation: the probability-weighted average

E[X] = Σ x · p(x)          (discrete)
E[X] = ∫ x · p(x) dx       (continuous)

Expectation is linear — E[aX + bY] = aE[X] + bE[Y] — which is why it survives inside gradients and losses so gracefully. Its companion, variance Var(X) = E[(X − E[X])²], measures spread: how wrong the average tends to be.

Why it matters. Training a neural network is minimizing an expected loss over the data distribution — we can never compute it exactly, so SGD estimates it with mini-batch averages (that's an expectation approximated by sampling). Cross-entropy loss, temperature sampling, top-p decoding, RLHF reward maximization — all are statements about distributions and expectations. When we cover loss functions and sampling later, they will literally be one-line formulas in this language.

Worked example. A tiny "language model" over 4 tokens, and the expected length of the chosen token:

import numpy as np

tokens = ["cat", "dog", "elephant", "ox"]
p      = np.array([0.62, 0.21, 0.12, 0.05])   # softmax output: sums to 1
length = np.array([3, 3, 8, 2])               # a value per outcome

E_len = (p * length).sum()                    # E[length] = Σ x·p(x)
print(E_len)                                  # 3.55

samples = np.random.choice(tokens, size=100_000, p=p)
print(np.mean([len(s) for s in samples]))     # ≈ 3.55 — sampling ≈ expectation

The Monte-Carlo average converges to the exact expectation as samples grow — the law of large numbers in action, and the exact trick SGD uses on every batch. Next up: conditional probability and Bayes' rule, the grammar of "given what I've seen so far".