For a while in 2024, a favorite way to trip up a chatbot was to ask it how many r’s are in “strawberry”. Plenty of them said two, confidently. These were models that could write working code and explain a tax form, beaten by a question a seven-year-old answers by pointing at each letter in turn.
Pointing at the letters is exactly what the model can’t do, because it never sees them. Before your question reaches the model, a separate program chops the text into chunks and swaps every chunk for a number. By the time “strawberry” arrives, it’s one number, or two or three, and the r’s are sealed inside.
That chopping step decides what a model can see, what a request costs and how much fits in its context window. It also explains several odd failures that otherwise look like the model being dim.
Text in, integers out
A language model is a neural network, and neural networks do arithmetic. Text has to become numbers before anything else can happen. The component that does this is the tokenizerTokenizerComponent that splits text into tokens and maps them to IDs (and back).. It splits text into tokensTokenA chunk of text (about ¾ of an English word on average) with an integer ID. The unit of processing and billing. and gives each token an integer ID.
Here’s the worked example from my notes. Take the text the unbelievable. A tokenizer might cut it into four tokens, "the", " un", "believ" and "able", and replace them with four IDs, say [1820, 653, 33467, 481]. (The split is illustrative and the IDs are made up: every tokenizer has its own.)
Notice that " un" starts with a space. In most modern tokenizers the space in front of a word belongs to the word’s first token, so the figures in this post draw it as a faint ·. Step through the example below, then switch it to strawberry and try counting the r’s from what the model gets.
The last step is the one that matters. The model receives [1820, 653, 33467, 481] and nothing else. (Through a chat API it also gets a few special tokens, added by the provider, that mark where each message starts and who wrote it; they’re billed too.) Each ID is only a row number in the tokenizer’s vocabulary, and the model never does arithmetic with it: 33467 isn’t “more” than 481, and nothing about 15717, the ID the figure gave berry, says b-e-r-r-y. Inside the model, each ID selects one row from a big table of learned numbers, the token’s embeddingEmbeddingA vector of numbers representing the meaning of a token or a piece of text. (Part 4 follows that row into the model, and Part 11 meets its cousin, the embedding of a whole text). Whatever the model knows about a token, including how it’s spelled, it had to pick up indirectly from patterns in its training text.
So “how many r’s in strawberry?” asks the model to count letters inside sealed chunks. It’s like asking someone to count the r’s in a word they’ve only ever heard spoken. They can do it if they happen to remember the spelling, but they can’t just look.
Why chunks, and not letters or words?
There are two obvious alternatives to chunks, and both are worse.
One token per character. The vocabulary would be tiny and nothing would ever be unknown. But every text would turn into four or five times as many tokens, and nearly everything about running a model scales with the token count: the compute per request, the memory, how much fits in the context window. The attention step in Part 4 compares tokens with each other, so its cost grows faster than the length. And since a model writes one token per run, it would write one letter per run.
One token per word. Sequences would be short, but the vocabulary would never end: every name, typo, URL, number, hashtag and word coined next year. A word missing from the list couldn’t be read (older systems swapped it for a catch-all “unknown” token), and the model could never write it, since it can only output tokens it has.
Modern tokenizers take the middle road. Their vocabulary holds the chunks of text that appear most often in the text the tokenizer was trained on. A common word like “the” gets a token to itself. A rarer word like “unbelievable” is assembled from pieces such as un, believ and able. And anything stranger, a typo, an emoji, a name in an alphabet the tokenizer barely saw, can fall back to smaller pieces, in byte-level tokenizers all the way down to single bytes. Tokens aren’t characters, and they’re not necessarily words.
On average an English token is about ¾ of a word, or roughly 4 characters. Hold on to the word English: that ratio belongs to production tokenizers on English text, and you’ll soon see how far it moves.
How a tokenizer learns its chunks
How does a tokenizer decide which chunks earn a place in its vocabulary? The most common method is Byte-Pair Encoding, or BPEBPEByte-Pair Encoding. Builds a token vocabulary by repeatedly merging the most frequent adjacent pairs., and it’s simple enough to do by hand:
- Split the training text into words. Each word keeps the space in front of it, and punctuation marks and runs of digits become chunks of their own. Pairs are never counted across these boundaries.
- Write each chunk out as its smallest possible pieces: single bytes (or characters).
- Count every pair of neighbors: how often
tis followed byh, how oftenhbye, and so on. - Take the most frequent pair, glue it into one new token, add that token to the vocabulary, and replace the pair everywhere in the text.
- Go back to step 3. Stop when the vocabulary reaches the size you chose in advance.
Because the space travels with the word that follows it, tokens like " un" start with a space.
Starting from bytes, not characters, is what makes byte-level BPE impossible to stump. Stored as UTF-8, any text in any language is a sequence of bytes, and a byte has only 256 possible values. So the 256 starting tokens can already encode anything at all; every merge after that is a shortcut for something common. Characters differ in size, though: an English letter is one byte, ę is two, and a typical Japanese character is three, so a tokenizer that never learned merges for them pays that many tokens per character.
To show the real thing, I wrote a 1,500-word piece about a small market town (a strawberry stall, a library, a church clock that runs seven minutes fast) and trained a tokenizer on it. The figure below runs the actual algorithm in your browser, merge by merge.
The natural guess is t + h, and if you took the hint about spaces you may have said space + t. Both are close: space + t comes second with 271, t + h third with 237. The winner is h + e, with 282, because he hides in “the”, “he”, “she”, “her”, “them”, “there”, “where” and “when”. (On a different text the top pairs can swap places: GPT-2’s tokenizer, trained on vastly more text, made space + t its very first merge.) Merge 2 glues the space onto t, and merge 3 joins ·t with he. Three merges in, the most common word in English is a single token, number 258.
The watch list shows the same process on rarer words. “Strawberry” appears often enough in this town’s story to become a single token at merge 194. And at merge 404, ·unbelievable settles into ·un + believ + able, the very split from my notes. Don’t read grammar into it: ·un and able earned their places by frequency, but believ comes from a pair the text contained only twice, both times inside “unbelievable”.
That’s the part worth dwelling on: splits follow frequency, not grammar. BPE has never heard of prefixes, stems or suffixes. It merges whatever is common. Sometimes the result looks like grammar, and sometimes it cuts straight through it. Drag the slider back to 75 and you’ll see eliev join the vocabulary, a fragment no dictionary lists, created because “believe” (9 times), “believes” (4) and “unbelievable” (2) all contain it. Between merges 172 and 193, “strawberry” is ·strawber + ry, sliced through the middle of “berry”. And ·un became a token at merge 189 because, by then, the pair ·u + n occurred six times: in “unbelievable” (twice), “unkind”, “unable”, “unreliable” and “university”. (“Under” starts the same way, three times over, but its n had been glued to the d back at merge 11, so it didn’t count. Merge order decides which words get counted at all.) BPE counts the un in “university” exactly like the prefix in “unkind”; it has no idea which is which.
The finished tokenizer is just two lists: the vocabulary, and the merges in the order they were learned. To tokenize new text, it splits the text into words, breaks each word into bytes, and replays the merges in order. GPT-style byte-level BPE tokenizers (GPT-2’s, GPT-4’s, Llama 3’s) do exactly this, trained on a vast and varied pile of text. Production vocabularies usually stop somewhere between 32,000 and 200,000 tokens.
How big should the vocabulary be?
Every merge makes text shorter. So why stop at 500 merges, or at 200,000 tokens? Why not merge until every word is a single token? Predict before you drag: if you double the number of merges, does the text get twice as short?
Not even close. Doubling the merges from 250 to 500 (a vocabulary of 506, then 756) takes the new paragraph from 265 tokens to 227: about 14% shorter, nowhere near half. The first 100 merges alone cut it from 620 tokens to 335; merges 400 to 500 shave off 9 more. Early merges capture pairs that occur everywhere; each later merge captures something rarer, so it saves less. From merge 391 on, every merge is a pair that occurred only twice in the whole training text, and by the end they are whole words like ·dragons and ·birthdays. Watch the two curves in that shaded region: the training text keeps getting shorter (2,612 → 2,255 tokens), while the paragraph the tokenizer has never seen improves only half as much (237 → 219), and only where it happens to contain one of those twice-seen words, like ·before or ·newspaper. Most of these merges memorize the training text instead of learning English, an artifact of training on 1,500 words: a production tokenizer learns from vastly more text, so even its last merges are pairs that occur over and over. What carries over is the shape: each merge saves less than the one before.
The cost, meanwhile, grows in a straight line. Every token in the vocabulary needs a row in two places. The embedding table holds the row of numbers the model looks up when the token comes in. The output layer holds the row the model uses to score that token as a possible next token, at every single step (the logitsLogitRaw, unnormalized score for each vocabulary token before softmax. from Part 1 come from there). Each row is as long as the list of numbers the model carries for every token from layer to layer: 4,096 in a typical 7–8B model. Some models save memory by using one table for both jobs (called weight tying), which halves this cost but doesn’t change its straight-line growth.
At full scale this is a real expense. Llama 2 7B has a 32,000-token vocabulary, rows of 4,096 numbers and no weight tying, so its two tables hold 2 × 32,000 × 4,096 ≈ 262 million parameters. Llama 3 8B keeps the same 4,096-number rows and quadruples the vocabulary to 128,256 tokens: 2 × 128,256 × 4,096 ≈ 1.05 billion parameters, spent on the vocabulary alone (Part 3 asks what share of the whole model that is). What the bigger vocabulary buys is shorter text: fewer tokens for the same words, which means cheaper calls and more text in the same context window. Meta reports that Llama 3’s tokenizer yields up to 15% fewer tokens than Llama 2’s: four times the vocabulary for at most 15% shorter text, the same diminishing returns as the toy. There’s a quieter cost, too. The bigger the vocabulary, the rarer its rarest tokens, and a token that barely appears in the model’s training data barely gets trained.
Use the tokenizer you just trained
Here is the 756-token vocabulary from Fig. 2, ready for anything you want to type. The presets cover the interesting cases.
Same meaning, very different bills: 17 tokens in English, 30 in Polish, 63 in Japanese. Polish isn’t a harder language. This tokenizer learned its chunks from English, so Polish words get almost no merges, and letters like ę and ś, which never appeared in training, fall all the way back to two raw bytes each. Each character in the Japanese greeting is three bytes in UTF-8, and none were ever merged, so its 21 characters cost exactly 63 tokens. The presets tell the same story in characters per token: 2.71 for an English paragraph the tokenizer never saw, 1.43 for the Python snippet, 1.23 for Polish, 0.33 for Japanese.
Production tokenizers do far better, because they have far bigger vocabularies trained on enormous, varied text, including many languages and lots of code. On English they average about 4 characters, or about ¾ of a word, per token. That second figure is the rule of thumb in my notes: 1,000 tokens ≈ 750 English words, or about 1.33 tokens per word.
The IDs are worth a look, too. ·the is token 258, ·The is 281, and The at the start of a line is 272. A lowercase the at the very start of the text isn’t in the vocabulary at all: it comes out as t + he, because in the training text “the” almost always follows a space. Every variant of a word is a different token with its own ID, and the model has to learn from data that they mean the same thing.
Now the second tokenizer. It was trained the same way, to the same size, on the same prose plus a page of Python. The code preset drops from 65 tokens to 41, because this tokenizer has tokens for def, ·return, and a line break followed by indentation. The English counts barely move, but nearly every ID changes: ·the is 259 here, and ·un is 496 instead of 444. Same algorithm, same size, different training text, different tokens.
Where tokens bite
Four everyday problems trace straight back to the chunking.
Counting letters
Back to strawberry. In the toy, ·strawberry is a single token, ID 449: one row of numbers, with no letters listed anywhere. Real models do learn a lot about spelling, because training text is full of spelled-out words, rhymes and dictionaries. But that knowledge is stored in the weights, not written on the page, and recalled facts are exactly what models get wrong with confidence.
Numbers that split oddly
Open the Numbers preset in Fig. 4. 1998 became ·1 + 9 + 98. 2024 became ·20 + 2 + 4. And 3,000 became ·, 3, ,, 00, 0. The comma is cut off by the pre-split rule from step 1; every other cut comes from which digit pairs happened to be common in the training text, with no regard for place value. To add two numbers, you line up units with units and tens with tens. A model that receives ·20|2|4 and ·1|9|98 has to work out where the units are, inside chunks of different sizes, from experience alone. It can learn this, but it’s a real handicap for arithmetic. Newer tokenizers tame it with rules: the tokenizer for GPT-4 cuts digit runs into groups of at most three, counting from the left, before merging, so 12345 becomes 123 + 45. That keeps chunks small but still doesn’t line up place value: the units digit ends a three-digit chunk in 123456 and stands alone in 1234567. Llama 1 and 2 sidestep the problem by splitting every number into single digits.
Paying more for the same meaning
Non-English text, code and unusual formatting (tables, long URLs, base64, heavy whitespace) usually take more tokens per word. In Fig. 4 the same greeting cost 17 tokens in English and 63 in Japanese. That costs you twice: you pay per token, and the context window fills faster, so a document that fits comfortably in English may not fit once translated. For a multilingual product, don’t budget with the English rule of thumb; measure real samples in each language.
Counting with the wrong tokenizer
The two toy tokenizers gave different counts and different IDs for the same text, and so do real models from different providers or generations. Token counts are model-specific: a budget computed with one model’s tokenizer is only an estimate for another, so always count with the tokenizer of the model you’re actually calling. Most providers publish theirs or offer a token-counting endpoint.
Paying by the token
Almost everything you buy from a language-model API is measured in tokens. Prices are quoted per million tokens, with separate rates for input (everything you send: instructions, history, documents) and output (everything the model writes). Rate limits are often set in tokens per minute. And the context window from Part 1 is a token budget.
The worked example from my notes uses hypothetical prices of $3 per million input tokens and $15 per million output tokens. A call with 20,000 input tokens and 1,000 output tokens costs:
Seven and a half cents for roughly 15,000 words of input, a long report, and a 750-word answer. Now look at the shape of that bill: output was under 5% of the tokens and 20% of the cost. Output tokens usually cost more than input tokens, often several times more. Part of the reason is the loop from Part 1. A token takes about the same arithmetic whichever side it’s on, but your input arrives all at once, so the hardware can push thousands of input tokens through in one parallel pass, reading the model’s weights from memory once for all of them. Output comes out one token per run, and every run has to read all of the weights again to produce just one new token per conversation. Providers share each read across a batch of conversations, but that’s still far more reading per token than a prompt needs. Part 3 puts numbers on the difference.
Before you touch the calculator, guess: for a long answer, 1,000 tokens in and 5,000 out, what share of the bill is output?
The Long answer preset has the answer. Output is 83% of the tokens and 96% of the bill. Big document flips it: 150,000 tokens in and a 500-token answer, and input is almost the entire cost. Which side is worth optimizing depends on the shape of your calls. For document-heavy work, send less: retrieve only the relevant passages (Part 12) instead of whole files. For chatty work, ask for shorter answers and cap the output length. And if the same long prefix goes out on every call, prompt caching (Part 1) usually bills the repeated part at a discount.
Each of those IDs is about to become a row of numbers, and numbers like those are all a model is made of. That’s next: weights, parameters, and what happens when a model actually runs.