3Blue1Brown neural networks ch. 5: transformers — notes and key slides

Updated 2026-10-02 · Video: 3Blue1Brown, published 2024-04-01 · 27 min

10 key slides

Watch the video (27 min)

Watch the original video on YouTube

This is Chapter 5 of 3Blue1Brown’s deep learning series and the first of three chapters on transformers, the architecture behind GPT and most current large language models. It follows the data through a GPT-style model, then slows down on the parts at the very start and very end of the network: tokens, embeddings, the unembedding matrix and softmax. You do not need the earlier chapters, but the idea that a network is a pile of tunable weights trained by gradient descent helps. If you want a short overview first, read LLMs explained briefly. The next chapter is attention.

Key takeaways

Generation is prediction on a loop. The model outputs a probability distribution over the next token; you sample one, append it, and run again (1:56).
Two kinds of block alternate. Attention lets vectors exchange information; the MLP block updates each vector on its own. Both are mostly matrix multiplications (3:59).
Weights and data are different things. The weights are learned and fixed after training; the data is whatever text you feed in. GPT-3’s 175 billion weights sit in 27,938 matrices of eight kinds (11:07).
Embedding directions carry meaning, and the dot product measures how well two vectors align. Attention, in the next chapter, is built on exactly this (16:39).
Softmax turns raw scores (logits) into probabilities, and temperature controls how adventurous sampling is (22:22).

Chapter notes

0:00 – 3:09 What “GPT” means and why next-word prediction can write

GPT stands for Generative Pretrained Transformer. “Pretrained” means the model first learned from a huge text corpus and can be fine-tuned later. The transformer is the specific network design that made the current wave of AI possible; the original 2017 version from Google was built for translation.

The model studied here does one job: given text, output a probability for every possible next token. That sounds far from writing an essay, but a loop closes the gap. Sample a token, append it, predict again. Grant runs this loop with GPT-2 on his laptop and gets a story that drifts into nonsense, then swaps in GPT-3, the same basic design at much larger scale, and gets something coherent (2:23). The point to keep: nothing in the loop changed, only the model size.

3:09 – 6:38 The high-level data flow

Four-panel overview slide labelled Embedding, Attention, MLPs and Unembedding, each with a small diagram — 3:09 — The roadmap for the whole transformer mini-series. Chapter 5 covers the two outer panels (embedding and unembedding); Chapters 6 and 7 cover the middle two.

The pipeline in order:

Tokenize. Text is cut into tokens: words, word pieces or punctuation. For images or audio, tokens can be patches or short chunks.
Embed. Each token becomes a long vector of numbers, its starting meaning.
Attention. Vectors pass information to each other so each one can absorb context.
MLP (feed-forward). Each vector goes through the same operation on its own, with no cross-talk.
Repeat attention and MLP many times (with normalization steps in between that the video skips).
Unembed. The last vector becomes a probability distribution over the vocabulary.

Two phrases side by side, 'A machine learning model' with a neural network picture and 'A fashion model' with a photo of a fashion model — 4:14 — The same token, "model", needs different meanings in different contexts. Fixing that is the job of attention: the embedding table alone cannot tell the two apart.

The MLP step is described as asking each vector a long list of questions and updating it based on the answers (4:48). Chapter 7, how LLMs store facts, makes that idea concrete.

A system prompt reading 'What follows is a conversation between a user and a helpful, very knowledgeable AI assistant' followed by a user question about Santiago, feeding into a Transformer box — 6:11 — How a text predictor becomes a chatbot: wrap the user's message in a system prompt that sets up a dialogue, then let the model predict the assistant's lines. The video notes that extra training is needed to make this work well.

6:38 – 12:39 The deep learning premise: weights, data and matrices

This is a refresher for anyone who skipped the earlier chapters. Machine learning means setting up a flexible model with tunable parameters and using examples to set them. Linear regression has two parameters; GPT-3 has about 175 billion.

Deep learning models share one training method, backpropagation (see backpropagation, intuitively), and that forces a format: inputs and layers are arrays of real numbers (tensors), and parameters interact with data only through weighted sums, usually packaged as matrix-vector products, with simple non-linear functions in between. Once you know this, many transformer design choices stop looking arbitrary. They are what you get when every step must be a differentiable matrix operation that runs fast on GPUs.

Table titled 'We will break this down' listing GPT-3's eight weight groups — Embedding, Key, Query, Value, Output, Up-projection, Down-projection, Unembedding — with the formula and count for each, totalling 175,181,291,520 — 11:45 — The running tally that spans Chapters 5–7. Each attention matrix group is about 14.5 billion weights and each MLP projection about 58 billion. Bookmark this frame; the next two chapters fill it in row by row.

Split slide: left side 'Weights — What defines the model' with blue and red matrices, right side 'Data — What the model processes' with grey vectors — 12:20 — The colour convention used for the rest of the series: blue and red for weights, grey for data. Keeping these apart is the single most useful habit when reading transformer diagrams.

12:39 – 19:52 Embeddings: words as directions

The model has a fixed vocabulary of 50,257 tokens in GPT-3. The embedding matrix W_E has one column per token, and each column is a 12,288-dimensional vector. Looking up a token means picking its column. That is 50,257 × 12,288, about 617 million weights.

The striking part is that training arranges these vectors so that directions mean something. Grant demonstrates with a small word-to-vector model (not a transformer): the difference between “woman” and “man” is close to the difference between “queen” and “king”.

3D plot with arrows for E(king), E(man) and E(woman), and the equation E(queen) ≈ E(king) + E(woman) − E(man) — 15:17 — The classic analogy. Grant points out it only holds approximately: in his model the true "queen" vector sits a bit off, because "queen" is not used in text merely as a female king.

A pitfall here: do not take these 3D pictures literally. They are projections of a 12,288-dimensional space onto a chosen 3D slice. The useful takeaway is that a high-dimensional space has room for many independent “meaning directions”, an idea that returns as superposition in Chapter 7.

Two arrows v and w on a grid next to the column-vector dot product v·w = −1.99, labelled Dot product — 16:39 — The dot product: positive when vectors point the same way, zero when perpendicular, negative when opposed. Attention scores in Chapter 6 are exactly these dot products.

Grant tests the idea with a “plurality direction” (cats minus cat): plural nouns score higher against it than singular ones, and “one, two, three, …” score increasingly high (17:17). He then stresses that in a transformer an embedding is only a starting point. As it flows through the network, a vector that began as “king” can come to encode which king, where and in what style (18:33).

19:52 – 22:22 Context size and the unembedding matrix

A 3D block of columns of numbers inside the transformer, with a brace labelled 'Context size' along its length — 19:52 — The network processes a fixed number of vectors at once. GPT-3 was trained with a context size of 2,048 tokens, which is why early chatbots seemed to lose the thread in long conversations.

At the end, an unembedding matrix W_U maps the last vector to one raw score per vocabulary token. It has the same shape as W_E, transposed, so another 617 million weights. In the Harry Potter example the context “least favourite teacher, Professor” should push “Snape” to the top.

The unembedding matrix multiplying the last context vector to give a column of scores per vocabulary word, then softmax giving probabilities with Snape at 0.78 — 21:46 — Unembedding then softmax. The raw scores in the middle column are the logits; the right column is the probability distribution the model samples from.

Why only the last vector? Grant explains that in training it is more efficient to have every position predict its own next token at once (21:30). One passage then gives thousands of training signals. This is also the reason for the masking you will meet in the attention chapter.

22:22 – 26:08 Softmax, temperature and logits

Matrix outputs can be negative, larger than 1, and do not sum to 1. Softmax fixes all three: raise e to each value, then divide by the total. Large inputs dominate, but similar values still share weight, and the output changes smoothly as inputs change. That smoothness matters because training needs derivatives.

Softmax with temperature diagram: input scores, the formula e^(x/T) over the sum, T = 2.29 on a slider, and output probabilities with one bar at 0.73 — 24:06 — Temperature T divides every logit before the exponential. Larger T flattens the bars; T near 0 gives all weight to the maximum. Try mentally setting T to 1 to recover plain softmax.

The GPT-3 demo makes the trade-off concrete: at temperature 0 the story is a predictable Goldilocks retelling; at a high temperature it starts more originally and then falls apart (24:34). Grant notes the API caps temperature at 2, which he calls an arbitrary product choice rather than a mathematical limit. The raw inputs to softmax are called logits (25:51).

Skip list

0:51 – 1:25 Examples of other transformer uses (speech, DALL-E images). Nice context, no new concepts.
6:38 – 7:24 Roadmap of the series and what is coming next.
6:53 – 12:39 The deep learning refresher. Skim it if you have watched Chapter 1 to Chapter 3, but keep the weights-versus-data slide at 12:20.
26:27 – end Note about the next chapter’s early release for supporters, and credits.

Glossary

Token — a chunk of input (word, word piece, punctuation, or an image patch) that gets its own vector.
Embedding matrix (W_E) — the lookup table that turns each token into its starting vector; 50,257 × 12,288 in GPT-3.
Embedding space — the high-dimensional space those vectors live in, where directions can encode meaning.
Context size — how many tokens the model processes at once; 2,048 for GPT-3.
Unembedding matrix (W_U) — maps the final vector to one score per vocabulary token.
Logits — the raw, unnormalised scores that go into softmax.
Softmax — turns a list of numbers into probabilities that are positive and sum to 1.
Temperature — a constant in softmax that controls how spread out the distribution is.

FAQ

What does the transformer chapter of the 3Blue1Brown series cover?

It gives a high-level tour of how a GPT-style transformer turns text into a next-token prediction, then explains the start and end of the network in detail: tokens, the embedding matrix, the unembedding matrix, softmax and temperature. Attention and MLP blocks are left for Chapters 6 and 7.

Why does a language model only use the last vector to predict the next token?

At generation time only the last position's vector is turned into a probability distribution. During training, every position predicts the token that follows it at the same time, which makes one passage act as many training examples.

What does temperature do in softmax?

Temperature divides the logits before the exponentials. A higher temperature flattens the distribution so less likely tokens get picked more often; a temperature near zero puts almost all weight on the single most likely token.

Where do GPT-3's 175 billion parameters come from?

The video lists eight groups of weight matrices: embedding, key, query, value, output, up-projection, down-projection and unembedding. Embedding and unembedding are about 617 million each; most of the rest sit in the attention and MLP blocks covered in the next two chapters.

Studying another course? Paste its YouTube or Bilibili link on the home page to request a handout — or make your own with the desktop app (coming soon).