📌 Daily Briefing

Recurrent Neural Networks — giving a network a memory

Intuition. A CNN assumes structure in space: nearby pixels belong together, and the same filter is useful everywhere in the image. A recurrent neural network makes the analogous assumption about time: the input is a sequence — words, sensor readings, audio frames — and the same transformation should apply at every step. A feed-forward network handed a sentence has no notion that "not" came before "good"; it sees a fixed-size bag of numbers. An RNN walks through the sequence one element at a time and carries a hidden state h forward, a running summary of everything seen so far. That hidden state is the memory.

The mechanism. At each time step t the network takes the current input x_t and the previous hidden state h_{t-1} and produces a new hidden state:

h_t = tanh(W_xh · x_t  +  W_hh · h_{t-1}  +  b_h)
y_t = W_hy · h_t + b_y

Three weight matrices, and — this is the key point — they are the same at every step. That is weight sharing across time, exactly as a CNN shares a filter across space. It means a single set of parameters handles sequences of any length, and it means the network learns a general rule ("how to update my summary given one more token") rather than a separate rule per position. Unrolled over T steps, an RNN is just a very deep feed-forward network with tied weights, so it is trained with ordinary backpropagation — the special case is called backpropagation through time (BPTT): run forward through the whole sequence, then propagate gradients backwards from step T to step 1, summing the gradient contributions into each shared matrix.

Why it matters — and where it breaks. The gradient flowing from step T back to step 1 passes through W_hh once per step, so it is multiplied by roughly the same factor T times. If the relevant eigenvalue of W_hh is below 1, the gradient shrinks geometrically and vanishes; above 1, it explodes. Practically: an RNN learns dependencies 5–10 steps back reasonably well and dependencies 100 steps back essentially not at all. Exploding gradients have a cheap fix (gradient clipping — rescale the gradient if its norm exceeds a threshold). Vanishing gradients do not, and that failure is precisely what LSTMs and GRUs were invented to solve, with gated additive paths that let information flow across many steps without repeated multiplication. That is tomorrow's topic.

Why you still care in 2026. Transformers replaced RNNs for language because self-attention connects any two positions in one hop instead of T, and because attention parallelises across the sequence while recurrence is inherently sequential. But the RNN framing never went away: the KV cache during autoregressive decoding is a growing hidden state, and modern state-space models (Mamba and relatives) are explicitly linear RNNs designed so the recurrence can be computed in parallel. Understanding h_t = f(h_{t-1}, x_t) is what makes those architectures legible rather than magic.

# A minimal RNN cell in NumPy — forward pass over one sequence
import numpy as np

H, D = 8, 4                       # hidden size, input size
W_xh = np.random.randn(H, D) * 0.1
W_hh = np.random.randn(H, H) * 0.1
b_h  = np.zeros(H)

def rnn_forward(xs):              # xs: list of T vectors, shape (D,)
    h = np.zeros(H)               # initial hidden state = "empty memory"
    states = []
    for x in xs:
        h = np.tanh(W_xh @ x + W_hh @ h + b_h)   # same weights every step
        states.append(h)
    return states

xs = [np.random.randn(D) for _ in range(12)]
states = rnn_forward(xs)
print(states[-1].shape)           # (8,) — one summary of the whole sequence

# Watch the vanishing-gradient factor: repeated multiplication by W_hh
print(np.linalg.norm(np.linalg.matrix_power(W_hh, 20)))   # ~0 -> gradient dies