3Blue1Brown neural networks ch. 7: how LLMs store facts — notes

Updated 2026-10-02 · Video: 3Blue1Brown, published 2024-08-31 · 23 min

10 key slides

Watch the video (23 min)

Watch the original video on YouTube

Chapter 7 of 3Blue1Brown’s deep learning series covers the other half of a transformer layer: the multilayer perceptron (MLP) block. The computation is simple, but what it does is hard to interpret, so the video builds a toy example of how an MLP could store the fact “Michael Jordan plays basketball”. It also finishes the GPT-3 parameter count and ends with superposition, a hypothesis from interpretability research. Watch transformers and attention first; this chapter assumes both.

Key takeaways

Facts seem to live in the MLP blocks, according to the DeepMind work cited, though the full mechanism is unsolved (0:48).
An MLP is linear, ReLU, linear, then add. Every token’s vector goes through it separately, with no cross-talk (1:37).
Rows of the first matrix ask questions; columns of the second say what to add. With a bias of −1, a row for “Michael + Jordan” fires only for the full name (8:38).
MLPs hold about two thirds of GPT-3’s parameters, around 116 billion, which completes the 175 billion tally (16:32).
Nearly perpendicular directions grow exponentially with dimension, so models may pack far more features than neurons, a superposition that makes single neurons hard to read (19:58).

Chapter notes

0:00 – 2:15 Where do facts live?

If a model completes “Michael Jordan plays the sport of” with “basketball”, that knowledge is somewhere in its weights. Grant cites a post by Google DeepMind researchers on matching athletes to sports. Their high-level finding is that facts appear to sit in the MLP blocks. He credits Neel Nanda, one of those researchers, for the layout of the example.

Slide with a Linear → ReLU → Linear diagram labelled 'Structure: Easy' and a grid of grey dots labelled 'Emergent behavior: Exceedingly…' — 1:37 — The tension of this chapter: the structure is trivial, the learned behaviour is not. Keep the toy example below in that light; it shows what is possible, not what models were measured doing.

2:15 – 6:22 Recap and the toy assumptions

The recap: tokens become vectors, vectors alternate between attention and MLP blocks, and directions in embedding space carry meaning. Attention has already pulled context in; MLPs are a candidate for adding stored knowledge.

The toy setup assumes three nearly perpendicular directions: “first name Michael”, “last name Jordan” and “basketball”. A vector encodes “first name Michael” if its dot product with that direction is 1. It also assumes an earlier attention head has already copied “Michael” into the vector at the “Jordan” position, because the name spans two tokens.

3D axes with a yellow arrow for 'First Name Michael', a red arrow for 'Last Name Jordan', a faint 'Basketball' direction and a white vector E combining the first two — 5:49 — The assumed geometry. A vector E encoding "Michael Jordan" has dot product 1 with both name directions. The MLP's job is to add the basketball direction when, and only when, that happens.

6:22 – 10:09 Step one: the up-projection and bias

MLP block diagram titled 'Michael Jordan plays Basketball': input vector, Linear, ReLU, Linear, then a plus sign adding the result back to the input — 7:07 — The whole block for one token: linear, ReLU, linear, then add the result to the vector that came in. That last addition (a residual connection) is easy to miss in the diagram.

A good way to read a matrix-vector product here is row by row: each row is a direction, and each output entry is that row’s dot product with the input. If row 0 equals M + J (Michael plus Jordan), the output is M·E + J·E.

Matrix with rows R0 to Rn multiplying vector E; R0 is labelled F.N. Michael + L.N. Jordan, and R0·E = (M + J)·E = M·E + J·E, approximately 2 if E encodes Michael Jordan — 8:38 — One row as one question. The score is about 2 for the full name and at most about 1 for "Michael Phelps" or "Alexis Jordan". Adding a bias of −1 moves the full-name case to 1 and the others to 0 or below.

In GPT-3 this matrix has 49,152 rows, exactly four times the embedding size, which Grant calls a hardware-friendly design choice (9:35). Because it maps into a larger space, he labels it W↑ (up-projection).

10:09 – 12:04 ReLU as an AND gate

A purely linear step cannot say a clean yes or no: a high score for Michael + Jordan also partly fires for half-matches. The fix is a non-linear function between the two matrices.

Vector passed through a Rectified Linear Unit, with the first entry kept at about 1 'if E encodes Michael Jordan' and everything zero or below clipped, next to the ReLU graph — 10:59 — ReLU clips negatives to zero. Combined with the −1 bias, this neuron outputs 1 for the full name and 0 otherwise: an AND gate built from a dot product and a clip.

Many models use GELU, a smoother version of the same shape. The entries after this function are what people mean by the “neurons” of a transformer; a neuron is active when its value is positive (11:27). This is the same layer type as the digit-recognition network from Chapter 1.

12:04 – 14:56 Step two: the down-projection

The second matrix maps back to embedding size, so Grant calls it W↓. Here it is easier to read column by column: the output is the sum of each column scaled by its neuron’s value.

Down-projection matrix shown as columns C0 to Cm multiplied by neuron values n0 to nm, written as n0 C0 + n1 C1 + n2 C2 + …, plus a bias vector B — 12:35 — Columns as "what to add". If column 0 is the basketball direction, it is added only when the Michael Jordan neuron is on. The column can carry other associated features too.

Put together: a vector that encodes Michael Jordan goes in, the AND-gate neuron fires, the basketball column is added, and the vector comes out encoding all three. Every token runs this in parallel, so GPT-3 has about 50,000 neurons per token per layer (14:16).

A useful way to hold this: the up-projection is a bank of detectors, the down-projection is a bank of responses. Attention decides what the vector knows about its context; the MLP looks at what the vector now is and adds what the model knows about it.

14:56 – 17:06 Finishing the parameter count

Full GPT-3 parameter table: Embedding and Unembedding 617,558,016 each, Key, Query, Value and Output 14,495,514,624 each, Up-projection and Down-projection 57,982,058,496 each, total 175,181,291,520 — 16:48 — The completed tally that started in Chapter 5. The two MLP rows dominate: about 116 billion of the 175 billion weights. Biases and normalization parameters are left out as a trivial share.

17:06 – 21:52 Is this how real models do it? Superposition

The two reading rules (rows as directions to detect, columns as directions to add) are just linear algebra and always true. What is doubtful is that one neuron stands for one clean feature like Michael Jordan. The evidence says that is rare.

The proposed reason is superposition. In N dimensions you can fit only N exactly perpendicular directions. But if you accept directions between 89° and 91° apart, the count changes dramatically in high dimensions. Grant runs a short Python experiment: 10,000 random 100-dimensional vectors, nudged by an optimizer to be as perpendicular as possible.

Python plot window showing a histogram of pairwise angles between vectors, sharply peaked around 90 degrees — 19:46 — After optimization, the angles between all pairs of 10,000 vectors in only 100 dimensions sit in a narrow band around 90°. That is 100 times more nearly independent directions than dimensions.

Slide: choose multiple vectors each pair between 89 and 91 degrees apart; the Johnson-Lindenstrauss lemma implies the maximum number of vectors is about exp(epsilon times N) in N-dimensional space — 19:58 — The Johnson–Lindenstrauss consequence: the number of nearly perpendicular directions grows exponentially with dimension. Grant suggests this may partly explain why model performance scales so well with size.

The same applies to the neuron layer. With about 50,000 neurons, an MLP could be probing many more features, each stored as a combination of neurons rather than a single one.

Neuron layer diagram with one neuron labelled Mirror and a faded label Pole Vaulting, next to a burst of nearly perpendicular arrows — 20:54 — If features are superimposed, a feature is a direction across many neurons, not one neuron lighting up. This is why researchers use sparse autoencoders to pull features back out.

Grant points to Anthropic’s posts on superposition and on sparse autoencoders as further reading (21:34).

21:52 – 22:22 What is still missing

Training is not covered here (21:52): the cost function for language models, fine-tuning with human feedback and scaling laws are left for a later chapter. The cost function part is the subject of the cross-entropy video.

Skip list

2:15 – 4:25 Recap of Chapters 5 and 6. Skip if they are fresh.
13:44 – 13:55 Aside on the down-projection bias, which the video says to ignore.
19:01 – 19:46 The live coding. Watch the result at 19:46; the setup is in our notes.
21:52 – end Plans for future videos and credits.

Glossary

MLP (multilayer perceptron) — the block of linear, non-linear, linear layers applied to each token separately; also called feed-forward.
Up-projection (W↑) — the first matrix, mapping 12,288 dimensions to 49,152 in GPT-3.
Bias — a learned vector added after a matrix multiplication.
ReLU — sets negative values to zero and keeps positive values unchanged.
GELU — a smoother version of ReLU used in many models.
Neuron — one entry of the vector after the non-linearity; active when positive.
Down-projection (W↓) — the second matrix, mapping back to embedding size.
Superposition — representing more features than dimensions by using nearly perpendicular directions.

FAQ

Where are facts stored in a large language model?

The video cites work by Google DeepMind researchers suggesting facts mostly live in the multilayer perceptron (MLP) blocks. A full mechanistic explanation is still unsolved, so the Michael Jordan example in the video is a toy model of how it could work, not a measured circuit.

What does an MLP block in a transformer compute?

Each token's vector is multiplied by a large up-projection matrix and a bias is added, a ReLU or GELU clips negatives to zero, then a down-projection matrix maps back to embedding size. The result is added to the original vector. Every token goes through this independently.

What is superposition in neural networks?

It is the hypothesis that a model represents many more features than it has dimensions by using directions that are nearly, not exactly, perpendicular. If true, single neurons rarely mean one clean thing, which is part of why models are hard to interpret.

How many of GPT-3's parameters are in MLP blocks?

About 116 billion of the 175 billion, roughly two thirds. Each up- and down-projection matrix is about 604 million weights, and there are 96 layers.

Studying another course? Paste its YouTube or Bilibili link on the home page to request a handout — or make your own with the desktop app (coming soon).