📌 Daily Briefing

Intuition. Suppose you flip a coin 10 times and see 7 heads. What's your best guess for the coin's bias p? MLE answers: pick the parameter value that makes the observed data most probable. Among all possible coins, the one with p = 0.7 assigns the highest probability to "7 heads in 10 flips" — so that's the maximum likelihood estimate.

Mechanism. Define the likelihood L(θ) = P(data | θ), viewed as a function of the parameters θ with the data fixed. For independent observations it's a product of per-example probabilities, so we maximize the log-likelihood instead — the log turns products into sums, is monotone (same argmax), and is numerically stable:

θ* = argmaxθ Σᵢ log P(xᵢ | θ)

For the coin: log L(p) = 7·log p + 3·log(1−p). Set the derivative to zero → 7/p = 3/(1−p) → p* = 7/10. Calculus (phase 1's gradients!) does the estimation.

Why it matters for AI. Training a neural network with cross-entropy loss is maximum likelihood: minimizing negative log-likelihood of the labels equals maximizing the probability the model assigns to the training data. Language-model pretraining — "predict the next token" — is MLE over text. Nearly every loss function you'll meet is an MLE (or a regularized/Bayesian cousin of one).

import numpy as np

# 10 coin flips, 7 heads
data = np.array([1,1,1,0,1,1,0,1,0,1])

def neg_log_likelihood(p):
    return -np.sum(data*np.log(p) + (1-data)*np.log(1-p))

ps = np.linspace(0.01, 0.99, 99)
best = ps[np.argmin([neg_log_likelihood(p) for p in ps])]
print(best)          # ≈ 0.70  — matches k/n = 7/10
# Cross-entropy loss in ML is exactly this NLL,
# just with a neural net producing p per example.

Learning progress

Phase 2 von 20 · 10 Konzepte gelernt

• 06-30 · Vectors & the dot product
• 07-01 · Matrices & matrix multiplication
• 07-02 · Derivatives & gradients
• 07-03 · The chain rule
• 07-06 · The Jacobian matrix
• 07-07 · Probability distributions & expectation
• 07-08 · The Hessian matrix
• 07-09 · Bayes' theorem
• 07-10 · Random variables, expectation & variance
• 08-03 · Maximum likelihood estimation ← today