3Blue1Brown neural networks ch. 6: attention — notes and key slides

Updated 2026-10-02 · Video: 3Blue1Brown, published 2024-04-07 · 26 min

10 key slides

Watch the video (26 min)

Watch the original video on YouTube

Chapter 6 of 3Blue1Brown’s deep learning series opens up the attention block, the part of a transformer that lets each token’s vector take in information from the rest of the text. It walks through a single attention head with a made-up example, then scales up to multi-head attention and counts GPT-3’s parameters. Watch Chapter 5 on transformers first: you need embeddings, dot products and softmax from there. Grant says himself that many people find attention confusing, so expect to rewatch parts. The next chapter is how LLMs store facts.

Key takeaways

Attention moves context into embeddings. The first embedding of “mole” is the same in every sentence; attention computes what to add so it points toward the right meaning (2:00).
Query-key dot products score relevance, and a column-wise softmax turns them into the attention pattern (8:30).
Masking keeps later tokens from influencing earlier ones, and the pattern grows with the square of the context size (12:14).
Values carry the actual update. Each column adds a weighted sum of value vectors to the original embedding (13:37).
Many heads run in parallel, 96 per block in GPT-3, and all attention adds up to about 58 billion parameters, a third of the model (24:16).

Chapter notes

0:00 – 2:00 Recap

A quick reminder of Chapter 5 (0:30): text becomes tokens, tokens become high-dimensional vectors, and directions in that space carry meaning. The transformer’s goal is to adjust those vectors step by step so they encode the meaning of a word in its context, not just the word.

2:00 – 4:34 What attention is supposed to do

Three phrases stacked vertically with the word mole highlighted: 'American shrew mole', 'One mole of carbon dioxide', 'Take a biopsy of the mole' — 2:00 — Three meanings of "mole", one starting vector. The embedding step is a lookup table with no context, so all three start identical. Attention is what separates them.

Grant’s mental picture: there are separate directions in embedding space for each meaning, and a well-trained attention block computes the step to add to the generic vector to land in the right one. “Tower” after “Eiffel” should move toward Paris and steel; add “miniature” and it should stop pointing toward large, tall things.

Attention can also move information over long distances. If the input is a whole mystery novel ending in “therefore the murderer was”, the last vector must somehow carry everything relevant to name the murderer (4:06). This connects back to Chapter 5: only the last vector is used for the prediction.

4:34 – 10:08 One head: queries, keys and the attention pattern

The working example is “a fluffy blue creature roamed the verdant forest”, with one imagined job: adjectives should update their nouns. Grant is clear that this is a made-up behaviour to make the matrices concrete. Real heads learn whatever helps prediction and are much harder to read.

The phrase 'a fluffy blue creature roamed the verdant forest' with embeddings E1 to E8; creature's embedding produces a 128-dimensional query asking 'Any adjectives in front of me?' while fluffy and blue reply 'I am!' — 6:30 — The query as a question. The query matrix W_Q maps each embedding into a smaller 128-dimensional space; for "creature" the imagined question is "any adjectives in front of me?"

A second matrix, W_K, maps every embedding into the same small space to produce keys. A key “answers” a query when the two vectors point the same way, so relevance is just their dot product.

Grid with queries along the top and keys down the side; large grey dots where the keys for fluffy and blue meet the query for creature — 8:48 — The grid of all key-query dot products. Big dots mark large scores: fluffy and blue "attend to" creature. Note the layout: keys are rows, queries are columns, so each column belongs to the token being updated.

The raw scores can be any real number. To use them as weights, each column goes through softmax so its entries are between 0 and 1 and sum to 1. The normalised grid is the attention pattern (10:08).

10:08 – 13:16 The formula, masking and the cost of context

The formula Attention(Q, K, V) = softmax(K^T Q / sqrt(d_k)) V above a table of every Q_i · K_j dot product — 10:41 — The formula from "Attention Is All You Need", decoded. K^T Q is the whole grid of dot products at once; √d_k is a scaling for numerical stability. Papers often write QK^T instead; the layout differs, the idea does not.

During training the model predicts the next token at every position simultaneously (Chapter 5 mentioned this). That creates a leak: if a later word could influence an earlier one, the model could see the answer. The fix is masking.

Unnormalized attention pattern with negative infinity in the lower-left entries, an arrow labelled softmax, and the normalized pattern where those entries are 0.00 — 12:14 — Masking. Setting those entries to zero directly would break the column sums, so they are set to −∞ before softmax, which turns them into exact zeros while keeping each column normalised.

Grant also points out that the pattern has one entry per pair of tokens, so it grows with the square of the context size (12:37). This is why long context windows are hard, and why research into sparse and blockwise attention variants exists. The video names a few but does not go into them.

13:16 – 16:47 Values: actually updating the embeddings

Knowing who is relevant is not enough; something has to be added. A third matrix, the value matrix, turns each embedding into a value vector that lives in the full embedding space. Think of it as “if I am relevant to you, here is what to add to you”.

The column under creature showing weights 0.42 on fluffy's value vector and 0.58 on blue's, summing to delta E4, which is added to E4 to give E4 prime, illustrated as a fluffy blue creature — 15:17 — The update for one column: weight each value vector by the attention pattern (0.42 for fluffy, 0.58 for blue here), sum to ΔE, add to the original embedding. The result should encode "a fluffy blue creature".

Every column does the same in parallel, so a whole sequence of refined embeddings comes out of the head. That is one head of attention, defined by three learned maps: query, key and value.

16:47 – 18:22 Parameter count and the low-rank value map

Query and key matrices are 128 × 12,288 each, about 1.5 million weights. A full value matrix would be 12,288 × 12,288, about 150 million, which is far more. In practice the value map is split into two thin matrices through the same small space.

Diagram of a low-rank transformation: a 12,288-dimensional vector mapped down to 128 dimensions by Value-down and back up to 12,288 by Value-up — 18:03 — The value map as a low-rank transformation: "value down" to 128 dimensions, then "value up" back to 12,288. Grant's names are non-standard; with four equal-sized matrices one head comes to about 6.3 million weights.

18:22 – 19:22 Side note: cross-attention

Cross-attention grid with English tokens 'I do not want to pet it' as queries and French tokens 'Je ne veux pas le caresser' as keys, with large dots linking corresponding words — 18:58 — Cross-attention: keys come from one sequence (French) and queries from another (English), so the pattern shows which words correspond. GPT models do not use it, and it needs no mask.

19:22 – 22:04 Multi-head attention

Adjectives updating nouns is only one kind of context. “They crashed the” changes what “car” means; “wizard” or “Queen, Sussex, William” changes which Harry is meant. Each kind of update needs its own key, query and value maps. So a block runs many heads in parallel, each with its own pattern, and adds all their proposed changes to each embedding.

Row of attention patterns for heads 1 to 4, each with its own W_K and W_Q, producing per-head changes delta E that are summed — 21:28 — Multi-head attention. Each head proposes a change ΔE for every position, and the block output is the original embedding plus the sum of all heads' changes. GPT-3 uses 96 heads per block.

22:04 – 24:54 Output matrix, layers and the 58 billion

One detail if you read papers or code: the “value up” matrices of all heads are usually stapled together into one output matrix for the whole block, and “the value matrix” of a head then means only the down-projection (22:47). It is the same computation organised differently.

Data also passes through many layers of attention and MLP. Each time, embeddings that are already richer get to absorb context from neighbours that are also richer, which is where higher-level features such as tone or topic can form.

GPT-3 parameter table with the Key, Query, Value and Output rows filled in at 150,994,944 per layer each — 24:16 — Running tally: about 151 million weights per matrix group per layer. Times 96 layers, attention totals just under 58 billion parameters, roughly a third of GPT-3. The rest is mostly MLPs (Chapter 7).

24:54 – 26:09 Why attention won

Grant’s closing point is not about any behaviour attention enables but about hardware: it is highly parallelisable, so it runs well on GPUs (24:54), and scale has been the main driver of better models. He recommends work by Andrej Karpathy and Chris Olah for going deeper.

Skip list

0:00 – 2:00 Recap of Chapter 5. Skip if you just watched it.
18:22 – 19:22 Cross-attention. Not used in GPT models; skip on a first pass.
22:04 – 23:37 Output-matrix convention. Only matters when you read papers or code.
25:28 – end Recommendations for other channels, and credits.

Glossary

Attention head — one set of query, key and value maps producing one attention pattern and one set of updates.
Query — a small vector representing what a token is looking for.
Key — a small vector representing what a token can offer; matched to queries by dot product.
Value — the vector a token contributes to another token’s embedding when relevant.
Attention pattern — the grid of softmax-normalised relevance scores between every pair of tokens.
Masking — setting future-to-past scores to −∞ before softmax so later tokens cannot affect earlier ones.
Multi-head attention — many heads run in parallel, with their updates summed.
Cross-attention — attention where keys and queries come from two different sequences.

FAQ

What are queries, keys and values in attention?

Each token's embedding is multiplied by three learned matrices. The query is what that token is looking for, the key is what it can offer, and the value is what it would add to another token's embedding if it turns out to be relevant.

Why does attention divide by the square root of the key dimension?

The video describes it as a numerical stability detail. Dot products of longer vectors tend to be larger, and scaling them down keeps the softmax from becoming too peaked.

What is masking in a GPT-style transformer?

Before softmax, every score where a later token would influence an earlier one is set to negative infinity, so it becomes zero after softmax while the columns still sum to one. This stops the model from seeing the answer it is supposed to predict during training.

How many parameters does attention use in GPT-3?

About 6.3 million per head, roughly 600 million per multi-head block with 96 heads, and just under 58 billion across all 96 layers. That is only about a third of GPT-3's 175 billion parameters; most of the rest sit in the MLP blocks.

Studying another course? Paste its YouTube or Bilibili link on the home page to request a handout — or make your own with the desktop app (coming soon).