📰 Daily Briefing

The perceptron and activation functions — where a linear model becomes a neural network

Intuition. Yesterday's linear regression computed y = w·x + b and stopped there. A perceptron takes exactly that same weighted sum and pushes it through one more step: a non-linear function σ that squashes or bends the output. So a single neuron is a = σ(w·x + b). That one extra function is the entire difference between "a line" and "a building block you can stack into something that learns arbitrary shapes."

Why the non-linearity is not optional. Stack two linear layers and you get W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂) — which is just another linear map. A hundred linear layers collapse into one. Depth buys you nothing without σ. Insert a non-linearity between layers and each layer can carve the input space with a new fold; the universal approximation theorem says a single hidden layer with enough such neurons can approximate any continuous function on a compact set arbitrarily well.

The mechanism. Geometrically, w·x + b = 0 defines a hyperplane. The weight vector w is its normal — it points in the direction of steepest increase — and b shifts the plane away from the origin. The pre-activation z = w·x + b is a signed, scaled distance from that plane. The activation then decides what to do with that distance. The original 1958 Rosenblatt perceptron used a hard step: fire if z > 0, else don't. That works, but the step function has zero gradient everywhere it is defined and an undefined gradient at zero, so it cannot be trained by gradient descent. That is the historical reason the field needed smooth activations.

The three activations worth knowing. Sigmoid, σ(z) = 1/(1+e−z), maps to (0,1) and has the pleasant derivative σ'(z) = σ(z)(1−σ(z)) — but that derivative peaks at 0.25 and vanishes in the tails, so gradients die in deep stacks. Tanh is a rescaled sigmoid centred at zero, which helps optimisation but still saturates. ReLU, max(0, z), has derivative exactly 1 for positive inputs — no saturation, no vanishing, and it is a single comparison to compute. That is why essentially every modern network uses ReLU or one of its smooth relatives (GELU, SiLU), and why deep learning became practical around 2012.

Why it matters for you. Every transformer you will meet is this pattern repeated: attention mixes information across positions, then a two-layer feed-forward block with a GELU in the middle does the actual per-token computation. Roughly two-thirds of the parameters in a typical LLM live in those perceptron-style MLP blocks. Understanding one neuron precisely is understanding most of the parameter mass.

import numpy as np

def relu(z):    return np.maximum(0.0, z)
def sigmoid(z): return 1.0 / (1.0 + np.exp(-z))

# One neuron solving XOR is impossible; two layers solve it.
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)

W1 = np.array([[1.0, 1.0],      # hidden neuron 1: "at least one input on"
               [1.0, 1.0]])     # hidden neuron 2: "both inputs on"
b1 = np.array([0.0, -1.0])
W2 = np.array([1.0, -2.0]);  b2 = 0.0

h = relu(X @ W1.T + b1)         # (4,2) hidden activations
y = sigmoid(h @ W2 + b2)        # (4,)  output

print(np.round(y, 3))           # [0.5 0.731 0.731 0.5]  -> XOR pattern

Note the collapse test: delete relu from line h = … and the network can no longer separate XOR at all, because it becomes a single linear map. That one line is the non-linearity argument, empirically.