Karpathy's Zero to Hero, Lecture 8: Let's build the GPT Tokenizer — notes, code and timestamps

Updated 2026-10-07 · Video: Andrej Karpathy, published 2024-02-20 · 134 min

77 key slides

1:193:065:566:1511:2412:5815:4816:4917:4218:5519:2421:0423:1125:3027:4229:4832:2833:4535:4337:5738:4640:2645:1645:5847:2348:1749:2751:4555:1256:5558:241:00:171:03:571:02:091:06:361:09:591:10:531:12:021:12:511:13:521:15:371:17:371:19:501:21:371:22:261:23:121:24:161:25:351:27:021:29:111:31:561:34:411:36:011:37:481:39:291:40:511:40:591:44:081:48:251:50:331:51:161:52:341:52:381:53:431:54:401:55:251:55:541:57:261:59:171:59:432:02:212:03:152:05:042:05:292:06:052:09:462:10:29

Lecture 8 of Andrej Karpathy’s Neural Networks: Zero to Hero series steps away from the model to look at the step that feeds it. In Lecture 7 the GPT read Shakespeare one character at a time, through a vocabulary of 65 characters. Production models don’t work that way: they read chunks of text produced by a trained tokenizer, usually built with byte pair encoding. Karpathy says up front that tokenization is his least favorite part of working with LLMs, and then spends two hours showing why it can’t be ignored. The lecture builds a byte-level BPE tokenizer from scratch in a notebook, compares it with what OpenAI ships for GPT-2 and GPT-4 and with Google’s SentencePiece (used by Llama 2), and closes by tracing a list of famous LLM failures back to tokens. The code from the lecture grew into the minbpe repository. Next in the series is Lecture 9, where the GPT-2 tokenizer covered here goes into a full model reproduction.

Key takeaways

1. A lot of LLM weirdness is tokenizer weirdness (4:40). Poor spelling, weaker non-English performance, shaky arithmetic and GPT-2’s trouble with Python all trace back, at least partly, to how text is cut into tokens.
2. Byte-level BPE is a short algorithm (26:18). Start from the 256 possible UTF-8 bytes, merge the most frequent adjacent pair into a new token, and repeat until you hit the vocabulary size you want.
3. The tokenizer is a separate model with its own training set (39:23). Its only parameters are the merges table and the vocabulary built from it, and the mix of languages and code in its training data decides how compactly each kind of text is encoded.
4. Not every token sequence is valid text (47:48). A model can emit bytes that aren’t valid UTF-8, so decoding uses errors="replace".
5. GPT tokenizers split text with a regex before BPE (1:03:56). Merges can’t cross the boundaries between letters, numbers, punctuation and whitespace; GPT-4’s pattern adds case-insensitive contractions, better whitespace handling and caps number chunks at three digits (1:13:44).
6. Special tokens skip BPE entirely (1:21:06). Strings like <|endoftext|> are swapped for reserved IDs, and adding new ones means resizing the model’s embedding table and output layer (1:24:43).
7. Vocabulary size is a trade-off, not a “bigger is better” knob (1:45:02). Larger vocabularies give shorter sequences but cost compute and leave rare tokens undertrained.
8. A token the model never trained on is a landmine (2:07:20). SolidGoldMagikarp shows what happens when the tokenizer’s data and the model’s data don’t match.

Chapter notes

0:02 – 4:08 Why tokenization, and where we left off

Karpathy opens by calling tokenization “fairly hairy” and full of hidden footguns. He recaps the Shakespeare notebook from the previous lecture (0:33): 65 unique characters, a lookup table from each character to an integer, and an embedding table with one trainable row per token. That row is how the Transformer “sees” a token.

Real models replace characters with chunks. He points to the GPT-2 paper’s section on input representation (2:36), which settles on a vocabulary of 50,257 tokens and a context of 1,024 tokens (3:07). Everything in an LLM is counted in tokens: the Llama 2 paper, he notes, mentions training on two trillion of them.

4:08 – 14:45 A tour of tokenizer weirdness

Before writing code, Karpathy lists symptoms that trace back to tokenization (4:40): trouble with spelling and simple string processing, worse results in non-English languages, bad arithmetic, GPT-2’s weakness at Python, warnings about trailing whitespace, the “SolidGoldMagikarp” meltdown, and advice to prefer YAML over JSON. He promises to come back to each one at the end.

Then he opens the Tiktokenizer web app (5:41), which tokenizes live in the browser, and walks through examples with the GPT-2 tokenizer:

• Spaces ride along with words. The leading space is part of the token, so “ is” and “ the” are single tokens.
• Numbers split arbitrarily (7:21). 127 is one token, 677 is two, and four-digit numbers break up in no consistent way.
• The same word gets many IDs (8:24). “Egg” at the start of a sentence is two tokens, “ egg” mid-sentence is one, and lowercase and uppercase versions are different tokens again. The model has to learn from data that these are all the same concept.
• Non-English text is stretched out (9:30). A Korean passage uses many more tokens than its English equivalent, partly because the tokenizer’s own training data was mostly English. Longer sequences mean the model runs out of context sooner.
• Python indentation is wasteful (11:03). In a FizzBuzz snippet, every indentation space is its own token (ID 220), eating context.

Switching the app to cl100k_base, the GPT-4 tokenizer, drops the same text from 300 tokens to 185 (12:36). The vocabulary roughly doubled, from about 50k to about 100k, and runs of spaces now merge into single tokens. Karpathy argues part of GPT-4’s better coding comes from this tokenizer design, not just the model. He also flags the cost: more tokens means a bigger embedding table and a bigger softmax at the output, so there’s a sweet spot.

14:45 – 23:39 From strings to code points to UTF-8 bytes

What is text in Python? A string is an immutable sequence of Unicode code points (15:49), integers the Unicode Consortium assigns to roughly 150,000 characters across 161 scripts. ord("h") returns 104 (16:51); an emoji returns a much bigger number.

Why not use code points directly as tokens? The vocabulary would be about 150,000 entries, and the standard keeps changing, so it isn’t a stable base (17:54). Encodings turn code points into bytes, and UTF-8 is the one to use (18:27): each code point becomes 1 to 4 bytes, and it’s backward compatible with ASCII. Encoding a string to UTF-16 and UTF-32 shows the waste: English text turns into rows of zero bytes (20:01).

Raw UTF-8 bytes alone give a vocabulary of only 256 (21:04). The embedding table would be tiny, but sequences would be very long, and attention over long sequences is expensive. Karpathy says he’d love to feed raw bytes straight into models, and shows a paper that proposes a hierarchical Transformer to do exactly that (22:37), but he doesn’t think it has been proven at scale yet. So the plan is to keep UTF-8 and compress the byte stream with BPE.

Our note: the byte-level paper on screen is MEGABYTE (Yu et al., 2023), whose abstract claims the viability of “tokenization-free autoregressive sequence modeling at scale”. The blog post Karpathy borrows text from throughout the notebook is Nathan Reed’s “A Programmer’s Introduction to Unicode”.

23:39 – 34:08 The BPE idea, then counting and merging pairs

The Wikipedia example (23:39) uses a four-letter alphabet. In the sequence aaabdaaabac, the pair aa is most common, so it becomes a new symbol Z. Then ab becomes Y, and finally ZY becomes X (24:41). Eleven symbols with a vocabulary of four turn into five symbols with a vocabulary of seven. Each round trades a slightly bigger vocabulary for a shorter sequence. With bytes, the same idea starts at 256 and mints token 256, 257, and so on (26:18).

The implementation starts on a paragraph of the Unicode blog post (26:48): 533 code points become 616 UTF-8 bytes, because non-ASCII characters take several bytes each. The first helper, get_stats, counts every adjacent pair (28:24). The top pair is (101, 32), which chr reveals to be “e” followed by a space, occurring 20 times: lots of words end in e (29:58).

max(stats, key=stats.get) picks the top pair (30:59), and a merge function walks the list left to right, replacing every occurrence with the new ID (31:30). The one trap is the bounds check at the last element. After merging, the list shrinks from 616 to 596, exactly 20 shorter, and no (101, 32) pair is left (33:37).

34:08 – 39:23 The training loop: a forest of merges

Repeating that step in a loop is the whole training procedure (34:08). How many times to repeat is a hyperparameter: more merges, bigger vocabulary, shorter sequences. Karpathy switches to the full blog post for better statistics and picks a final vocabulary of 276, which means exactly 20 merges (35:44).

The merges dictionary records each (child, child) → new token mapping. He describes the result as a forest rather than a tree (36:46): the 256 bytes are the leaves, and newly minted tokens can themselves be merged again, as the 20th merge shows when it combines two earlier merged tokens (38:19). With only 20 merges, about 24,000 bytes shrink to about 19,000 tokens, a compression ratio of roughly 1.27 (38:51).

39:23 – 42:32 The tokenizer is its own model

This is the conceptual point of the lecture (39:23). The tokenizer is a preprocessing stage, trained once, on its own training set, with BPE. Once trained it’s a translation layer between raw text and token sequences, in both directions. The LLM is trained afterwards, often on data that has been tokenized once and stored on disk as a giant token stream, with the raw text thrown away (40:57).

Because the tokenizer has its own data, its data mix matters (41:30). Lots of Japanese in the tokenizer’s training set means more Japanese merges, shorter Japanese sequences, and more Japanese text per context window for the LLM.

42:32 – 48:21 Decoding, and invalid UTF-8

Decoding goes from IDs back to a Python string (42:32). Build a vocab dictionary from token ID to bytes: the first 256 entries are single bytes, and each merged token is the concatenation of its two children’s bytes (43:33). This only works if the merges are visited in insertion order, which Python dicts guarantee from version 3.7 (44:06). Then join the bytes and call .decode("utf-8").

There’s a bug waiting (45:07). Decoding token 97 gives “a”, but token 128 alone raises a UnicodeDecodeError: 128 is 10000000 in binary, which isn’t a legal start byte in UTF-8’s multi-byte scheme. The fix is errors="replace" (46:44), which substitutes the replacement character (�). OpenAI’s released code does the same. If you see that character in model output, the model produced a token sequence that isn’t valid UTF-8 (47:48).

48:21 – 57:08 Encoding by replaying merges in order

Encoding goes the other way (48:21). Turn the text into UTF-8 bytes, then apply merges. Order matters: merges learned later can depend on tokens created by earlier ones, so earlier merges have to happen first (49:23).

Karpathy’s loop reuses get_stats, but only for its keys, the set of adjacent pairs currently present. Among those, it picks the pair with the lowest index in merges, using min with merges.get(p, float("inf")) as the key (50:57). Pairs that were never learned get infinity and can’t win. When nothing is mergeable, every pair scores infinity and min returns an arbitrary pair, so the loop checks whether that pair is in merges and breaks if not (53:01).

An edge case: a one-character or empty string has no pairs, which crashes min, so the loop runs only while there are at least two tokens (55:04). Tests close the section (56:07): encode-then-decode returns the original text, both on the training text and on unseen validation text. The reverse, decode-then-encode, isn’t guaranteed, since not every token sequence is valid UTF-8.

57:08 – 1:07:32 GPT-2’s regex pre-split

Real tokenizers add rules on top of plain BPE. The GPT-2 paper explains why (58:09): “dog” appears next to every kind of punctuation, and naive BPE would mint tokens like “dog.” and “dog!”, mixing meaning with punctuation. OpenAI’s fix lives in encoder.py in the GPT-2 repo (59:12), a regex pattern compiled with the regex package, a more powerful extension of Python’s built-in re (1:00:14).

re.findall with that pattern chops the text into chunks first (1:00:47). Each chunk is tokenized on its own and the results are concatenated, so a merge can never cross a chunk boundary (1:03:24). In practice the “e” and the space after it can no longer merge, because the space now starts the next word’s chunk (1:03:56).

Reading the pattern alternative by alternative:

The contraction rule is brittle. A curly Unicode apostrophe isn’t matched, and because OpenAI didn’t make the match case-insensitive, “HOW’S” splits differently from “how’s”, something their own code comment admits (1:06:29). Karpathy adds that hard-coding English contractions is probably unfair to other languages.

1:07:32 – 1:11:38 Whitespace, and rules nobody published

The punctuation alternative catches anything that isn’t a letter, number or space (1:07:32). The whitespace rule is the subtle one (1:08:02). Its negative lookahead takes a run of spaces up to, but not including, the last one, so that last space can attach to the following word. That keeps “ you” as the familiar single token even after extra spaces.

Running the pattern on a snippet of Python shows it splitting every time the character category changes (1:09:33). But that isn’t the whole story: in the real GPT-2 tokenizer, runs of spaces never merge at all, each staying token 220 (1:10:35). So OpenAI applied some extra rule during training. Nobody outside knows what it was, because only the inference code was released, never the training code (1:11:06).

1:11:38 – 1:18:28 tiktoken, the GPT-4 pattern and OpenAI’s encoder.py

OpenAI’s official tokenization library is tiktoken, again inference only (1:11:38). Encoding the same string with the gpt2 and cl100k_base encodings shows GPT-4 merging the whitespace that GPT-2 leaves alone. The patterns themselves are published in tiktoken’s openai_public.py (1:12:41), because inference needs them. The GPT-2 one there is a faster but equivalent form of the pattern above.

The GPT-4 pattern changes three things that matter (1:13:44): contractions now match case-insensitively, whitespace is handled differently, and numbers are matched in runs of one to three digits, so no token is ever longer than three digits (1:14:16). Why those choices? Undocumented. Karpathy suggests stepping through the regex with ChatGPT and the docs rather than doing it on camera.

Back in GPT-2’s encoder.py (1:14:49), two files make up the saved tokenizer. encoder.json is the equivalent of the lecture’s vocab (inverted, mapping string to integer), and vocab.bpe is, confusingly, the equivalent of merges (1:15:51). OpenAI also wraps the tokenizer in an extra byte_encoder / byte_decoder layer that Karpathy dismisses as an unimportant implementation detail (1:16:24). Strip that away and the bpe function is the same find-the-bigram, merge-it-everywhere loop built earlier (1:17:25).

1:18:28 – 1:25:13 Special tokens

GPT-2’s vocabulary has 50,257 entries (1:19:01): 256 bytes, 50,000 merges, and one more. That last one is <|endoftext|>, token 50256, inserted between documents in the training stream (1:19:32). The model has to learn from data that it means “what follows is unrelated”.

In Tiktokenizer, typing the string piece by piece produces ordinary tokens until it’s complete, then it snaps into the single ID 50256 (1:21:06). That’s because special tokens never go through BPE; tiktoken’s Rust core looks for them and swaps them in directly (1:21:36). Chat models lean on them heavily: the GPT-3.5-turbo format wraps every message in tokens such as <|im_start|> and <|im_end|> (1:22:38). tiktoken lets you fork cl100k_base and register your own (1:23:10), and GPT-4’s encoding adds fill-in-the-middle tokens (<|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>) plus one more (1:23:41).

Adding a special token to a trained model takes “model surgery” (1:24:43): one new row in the embedding table, usually initialized small and random, and one more output in the final projection. It’s routine when turning a base model into a chat model.

Our note: the extra cl100k_base token is <|endofprompt|>. Fill-in-the-middle comes from OpenAI’s 2022 paper “Efficient Training of Language Models to Fill in the Middle”.

1:25:13 – 1:28:51 minbpe and the exercise

Karpathy introduces minbpe, the repository that came out of preparing this lecture (1:25:13), and its exercise.md, which breaks building a GPT-4-style tokenizer into four steps (1:25:45). The target is to reproduce tiktoken’s GPT-4 encodings exactly, and also to write the training function tiktoken doesn’t have (1:26:48).

He compares GPT-4’s merge order with a minbpe tokenizer trained on the Taylor Swift Wikipedia article, chosen because it’s one of the longest (1:27:19). GPT-4’s very first merge was two spaces into one token, which hints at a lot of code in its tokenizer data. Otherwise the two lists look alike, as you’d expect from the same algorithm on different data.

1:28:51 – 1:35:34 SentencePiece: BPE on code points

SentencePiece, used by Llama and Mistral, can both train and run tokenizers (1:28:51). The big difference is the order of operations (1:29:22). tiktoken encodes to UTF-8 and runs BPE on bytes. SentencePiece runs BPE on code points directly, and handles rare code points (as defined by the character_coverage setting) either by mapping them to an unknown token or, with byte_fallback on, by encoding them as UTF-8 bytes that have their own byte tokens (1:29:53). Karpathy finds the tiktoken way cleaner.

The demo trains on a toy text file (1:30:54), and the first surprise is the sheer number of options, many irrelevant to BPE, which Karpathy puts down to accumulated history (1:31:25). He copied the settings Meta used for Llama 2 by opening Meta’s released tokenizer model file with protobuf (1:32:27). Two of his complaints:

• Normalization (1:32:58). Older NLP lowercased text and collapsed whitespace. For LLMs he prefers to leave the raw data untouched, so these rules are switched off.
• Sentences (1:33:28). SentencePiece treats sentences as training examples, with limits on their count and length. Karpathy argues “sentence” is ill-defined in raw internet text and would rather treat a file as one stream.

There are also options for splitting digits and whitespace, roughly SentencePiece’s version of the regex pre-split, and hard-coded unknown, beginning and end of sentence, and padding tokens.

1:35:34 – 1:43:29 Inside a Llama 2–style SentencePiece model

Training with a vocabulary of 400 produces a model whose vocabulary is laid out in a fixed order (1:35:34): special tokens first (<unk> is 0, beginning and end of sequence are 1 and 2, padding disabled), then the 256 byte tokens, then the merged tokens, then the individual code-point tokens seen in training (1:36:07).

Encoding “hello” followed by Korean text shows byte fallback at work (1:38:14). The Korean characters weren’t in the toy training data, so they come out as UTF-8 byte tokens, offset by 3 because of the special tokens at the front. Turning byte_fallback off (1:38:45) frees vocabulary space for more merges, but the whole Korean string collapses into <unk>, ID 0. Feeding a model one token for “everything I don’t recognise” is a bad idea, so Llama was right to turn fallback on (1:39:48).

Two more quirks. Spaces are displayed as a bold underscore character (▁). And there’s a space before “hello” that wasn’t in the input, added by add_dummy_prefix (1:40:21). In tiktoken, “world” at the start of text and “ world” mid-sentence are different tokens; the dummy prefix makes both “ world” so the model doesn’t have to learn they’re the same (1:41:54). Llama 2 uses it.

Karpathy’s verdict (1:42:27): widely used and efficient, but loaded with historical baggage, confusing concepts and thin documentation.

1:43:29 – 1:51:48 Choosing vocab size, adding tokens, and beyond text

Back in the GPT from Lecture 7, vocab_size appears in only two places (1:43:29): the rows of the token embedding table and the outputs of the final lm_head linear layer that produces the logits. Why not make it huge (1:45:02)?

In Karpathy’s telling, current models land in the high tens of thousands to around 100,000 (1:46:36). Extending a trained model’s vocabulary, for chat formatting or tool use, is the same mild surgery as before. You can even freeze the base model and train only the new rows (1:47:39).

He then points at the wider design space (1:48:10). The gist tokens paper compresses a long prompt into a few new tokens by training only their embeddings, distilling the model’s behavior with the full prompt. And for images, video and audio, the trend he sees is to keep the Transformer and tokenize the new modality instead (1:50:14), with hard (integer) or soft tokens, as in OpenAI’s Sora, which describes its “visual patches” as the counterpart of text tokens (1:50:45).

Our note: the paper is “Learning to Compress Prompts with Gist Tokens” (Mu, Li and Goodman, 2023).

1:51:48 – 1:56:58 Revisiting the quirks: spelling, languages, arithmetic, Python

Now the opening list gets its explanations (1:51:48):

• Spelling. .DefaultCellStyle is one GPT-4 token. Asked how many l’s it contains, GPT-4 says three; the answer is four (1:52:19). Asked to reverse it, GPT-4 produces a jumble (1:52:51). Asked to first list the characters separated by spaces and then reverse the list, it succeeds, because each character is now its own token (1:53:21).
• Non-English. “Hello how are you” is 5 tokens; its Korean translation is 15, and even the everyday Korean greeting takes 3 tokens where “hello” takes 1 (1:54:24).
• Arithmetic. Addition is a digit-by-digit algorithm, but digits are grouped arbitrarily (1:54:55). A blog post titled “Integer tokenization is insane” maps how four-digit numbers split in GPT-2 (1:55:25). Llama 2 splits all digits individually, partly for this reason.
• Python. GPT-2’s one-token-per-space indentation was close to a tokenizer bug, fixed in GPT-4 (1:56:27).

1:56:58 – 2:04:44 Special-token injection, trailing spaces and partial tokens

Ask GPT-4 to print the string <|endoftext|> and it can’t seem to see it (1:56:58). Karpathy doesn’t know what OpenAI does internally, but suspects user text is being parsed with special tokens allowed. User input is attacker-controlled, so special-token handling becomes an attack surface (1:58:01).

The trailing whitespace warning (1:58:31) appears in the OpenAI playground with the GPT-3.5-turbo-instruct completion model. Because GPT tokens carry their leading space (” o” is one token), a prompt ending in a bare space puts token 220 where the model expects the space to be part of the next word (2:00:05). That’s rare in training data, so the completion is out of distribution.

The same goes for cutting a token in half. Prompting with .DefaultCellSta (without the “yle”) makes the model end its completion immediately, and a retry gets flagged as a possible policy violation (2:01:40). Karpathy calls these “partial token” problems. tiktoken’s Rust code has a lot of undocumented handling for “unstable tokens” (2:03:44); ideally a completion API would search over continuations at the character level rather than append whole tokens.

2:04:44 – 2:09:26 SolidGoldMagikarp

Karpathy’s favorite (2:04:44). A blog post clustered GPT token embeddings and found a cluster of strange tokens: “SolidGoldMagikarp”, “StreamerBot” and other odd strings (2:05:16). Asking the model to simply repeat one of them produces evasion, hallucination, or insults (2:06:20).

The likely explanation (2:07:20): SolidGoldMagikarp is a Reddit user who posted a lot. The tokenizer’s training data included plenty of Reddit, so the username earned its own token. The language model’s training data apparently didn’t, so that token never appeared during training and its embedding row stayed at its random initialization (2:08:24). Karpathy compares it to unallocated memory in a C program: invoking the token plucks out an untrained vector and the model’s behavior is undefined.

Our note: the original post is “SolidGoldMagikarp (plus, prompt generation)” by Jessica Rumbelow and Matthew Watkins on LessWrong (2023). It’s also a direct, real-world case of takeaway 3: the tokenizer and the model were trained on different data.

2:09:26 – end Formats, final advice, and outtakes

Token efficiency depends on format (2:09:26). The same data is 116 tokens as JSON and 99 as YAML. Since you pay per token, in both context and money, it’s worth measuring formats in Tiktokenizer.

Closing advice (2:10:28): don’t brush tokenization off; it carries security and AI safety issues. And “eternal glory goes to anyone who can get rid of it”. His practical recommendations (2:10:59):

1. If you can, reuse the GPT-4 vocabulary with tiktoken.
2. If you must train your own vocabulary today, use BPE in SentencePiece, but copy Meta’s settings carefully, since misconfiguring it is easy (2:11:30).
3. What he really wants is tiktoken with training code. minbpe is a Python implementation of that idea, not yet efficient (2:12:02).

The final half-minute after the sign-off is outtakes.

Code you’ll write

The lecture builds everything in a notebook; minbpe packages the same logic into BasicTokenizer, RegexTokenizer and GPT4Tokenizer classes. These are short excerpts.

Count pairs and merge one (28:24, 31:30). get_stats tallies adjacent pairs; merge replaces every occurrence of one pair with a new ID.

def get_stats(ids):
    counts = {}
    for pair in zip(ids, ids[1:]):  # consecutive elements
        counts[pair] = counts.get(pair, 0) + 1
    return counts

def merge(ids, pair, idx):
    newids = []
    i = 0
    while i < len(ids):
        if i < len(ids) - 1 and ids[i] == pair[0] and ids[i+1] == pair[1]:
            newids.append(idx)
            i += 2
        else:
            newids.append(ids[i])
            i += 1
    return newids

Train: repeat the merge (35:44). Twenty merges take the vocabulary from 256 to 276, and merges records the order.

vocab_size = 276  # 256 raw bytes + 20 merges
num_merges = vocab_size - 256
ids = list(tokens)  # tokens = list(text.encode("utf-8"))
merges = {}  # (int, int) -> int
for i in range(num_merges):
    stats = get_stats(ids)
    pair = max(stats, key=stats.get)
    idx = 256 + i
    print(f"merging {pair} into a new token {idx}")
    ids = merge(ids, pair, idx)
    merges[pair] = idx

Decode (43:33). Build token-to-bytes from the merges, join, and decode leniently.

vocab = {idx: bytes([idx]) for idx in range(256)}
for (p0, p1), idx in merges.items():  # relies on insertion order
    vocab[idx] = vocab[p0] + vocab[p1]

def decode(ids):
    tokens = b"".join(vocab[idx] for idx in ids)
    text = tokens.decode("utf-8", errors="replace")
    return text

Encode (50:57). Always apply the earliest-learned merge available, and stop when none is left.

def encode(text):
    tokens = list(text.encode("utf-8"))
    while len(tokens) >= 2:
        stats = get_stats(tokens)
        pair = min(stats, key=lambda p: merges.get(p, float("inf")))
        if pair not in merges:
            break  # nothing else can be merged
        idx = merges[pair]
        tokens = merge(tokens, pair, idx)
    return tokens

GPT-2’s pre-split, and tiktoken for comparison (1:00:47, 1:11:38). The regex chunks text so merges never cross categories; tiktoken runs the real GPT-2 and GPT-4 tokenizers.

import regex as re
gpt2pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")
print(re.findall(gpt2pat, "Hello've world123 how's are you!!!?"))

import tiktoken
enc = tiktoken.get_encoding("gpt2")
print(enc.encode("    hello world!!!"))  # spaces stay separate
enc = tiktoken.get_encoding("cl100k_base")  # GPT-4
print(enc.encode("    hello world!!!"))  # spaces merge

A Llama 2–style SentencePiece run (1:32:27). A trimmed set of the options Karpathy copied from Meta’s tokenizer; the notebook sets several more.

import sentencepiece as spm
spm.SentencePieceTrainer.train(
    input="toy.txt", model_prefix="tok400",
    model_type="bpe", vocab_size=400,
    normalization_rule_name="identity", remove_extra_whitespaces=False,
    split_digits=True, byte_fallback=True,
    add_dummy_prefix=True, character_coverage=0.99995,
    unk_id=0, bos_id=1, eos_id=2, pad_id=-1,
)
sp = spm.SentencePieceProcessor()
sp.load("tok400.model")
ids = sp.encode("hello 안녕하세요")
print([sp.id_to_piece(i) for i in ids])

Full code: karpathy/minbpe (MIT license), including the lecture notebook, exercise.md and tests that check the GPT-4 tokenizer against tiktoken.

Skip list

• 0:02 – 2:36 Recap of the character-level tokenizer. Skip if you’ve just watched Lecture 7.
• 18:58 – 20:01 Pros and cons of UTF-16 versus UTF-32 and the reading list. The demo right after makes the point faster.
• 1:16:24 – 1:17:25 OpenAI’s byte_encoder / byte_decoder layer. Karpathy himself calls it an uninteresting detail and skips it.
• 1:31:25 – 1:35:02 The tour of SentencePiece configuration and the critique of “sentences”. Skip unless you plan to train with SentencePiece.
• 1:48:10 – 1:51:48 Gist tokens and multimodal tokenization. These are quick pointers to papers, with no code.
• 2:12:33 – end Outtakes after the sign-off.

Glossary

• Token — the unit an LLM actually reads and predicts: an integer ID that stands for a chunk of text, anywhere from one byte to a long word.
• Tokenizer — a separately trained component that translates text into token IDs and back; for BPE, it’s fully described by its merges and vocabulary.
• Unicode code point — the integer Unicode assigns to a character; what ord() returns for a one-character Python string.
• UTF-8 — a variable-length encoding that turns each code point into 1 to 4 bytes; ASCII-compatible and the basis of byte-level BPE.
• Byte pair encoding (BPE) — a compression algorithm that repeatedly replaces the most frequent adjacent pair of symbols with a new symbol; used to build LLM vocabularies.
• Merges — the ordered table of (token, token) → new token rules learned in training; encoding replays them earliest first.
• Vocab — the map from each token ID to the bytes it represents, built from the 256 single bytes plus the merges.
• Compression ratio — bytes in divided by tokens out; about 1.27 after 20 merges in the lecture.
• Regex pre-split (pre-tokenization) — chopping text into chunks of letters, numbers, punctuation and whitespace before BPE, so merges never cross those boundaries.
• Special token — a reserved ID such as <|endoftext|> or <|im_start|> that is inserted directly rather than produced by merges, used to mark documents or chat structure.
• Model surgery — resizing a trained model’s embedding table and output layer to add new tokens.
• tiktoken — OpenAI’s fast, inference-only BPE library containing the GPT-2 (gpt2) and GPT-4 (cl100k_base) encodings.
• SentencePiece — Google’s library for training and running tokenizers; runs BPE on code points and is used by Llama and Mistral.
• Byte fallback — a SentencePiece option that encodes code points missing from the vocabulary as UTF-8 byte tokens instead of <unk>.
• add_dummy_prefix — a SentencePiece option that puts a space in front of the text so a word at the start looks the same as mid-sentence.
• Partial (unstable) token — a prompt that ends partway through what would normally be one token, such as a trailing space, leaving the model out of distribution.
• Glitch token — a token present in the vocabulary but absent from the model’s training data, so its embedding is untrained; SolidGoldMagikarp is the classic example.

Studying another course? Paste its YouTube or Bilibili link on the home page to request a handout — or make your own with the desktop app (coming soon).