Pre-training: Learning Everything by Guessing the Next Word

Fill in the blanks before you read on. The capital of Australia is ___. Three tickets at $12 each come to $___. def is_even(n): return n % 2 == ___ By the end of the eulogy, there wasn’t a dry eye in the ___.

You just used a fact, some arithmetic, a bit of Python and a feel for the mood of a room, and all anyone asked of you was the next word. That’s the bet behind pre-training. The obvious way to teach a computer all that is a textbook and an exam, and it falls apart at once. Nobody can write down the rules of English, never mind English plus Python plus chemistry, and an exam big enough to cover them all would need billions of hand-written questions.

So nobody tries. Pre-training does something much lazier, and it works. Take any text at all. Cover up the next word. Ask the model to guess it, then uncover the word and tell the model how wrong it was. Do that a few trillion times. That one game, with no rules and no human-written answer key, is where most of what a large language model knows comes from. To see why it works, start much smaller than language: with a single number.

One number, nudged into place

In Part 3, a model was a pile of numbers called weightsWeightsThe learned numbers that scale inputs inside the network. Set during training, frozen at inference., and the smallest possible model had exactly one. It predicts a house’s price as w × floor area. With w = 300, a 1,000 sq ft house comes out at $300,000.

But where did 300 come from? Nobody typed it in. Training is how weights get their values, and it starts from a bad guess. In a real network the starting weights are random numbers, so the first predictions are useless. Here, start the one weight at w = 50 and show the model a fresh set of twelve sales (not Part 3’s), one at a time. For the first house: guess 1,000 × 50 = $50,000. Compare: it sold for $300,000, so the guess is $250,000 too low. Nudge w up a little, then try the next house.

Before you press play, guess two things. After a dozen sales, where will w be? And will the nudges get bigger or smaller as it goes?

Mostly smaller, but not always. Each nudge is the miss times the house’s size times a small fixed number (the rule comes from Part 6), so the 1,800 sq ft house at sale 4 still earns a big push. As the guesses close in, the misses shrink and the nudges shrink with them. After twelve sales w is at 299.1, right on the slope that fits these twelve sales best (for Part 3’s messier twelve it was 298). Not exactly 300: real houses don’t sit on one perfect line, and each sale pulls w toward its own price per square foot, so with more sales w would keep wobbling around 299. Which way to push each of billions of weights, and how much each one is to blame for the miss, is what backpropagationBackpropagationAlgorithm that computes how much each weight contributed to the error, using the chain rule, working from output to input. works out in Part 6; how far to actually step is the optimizer’s call. Part 6 works through this exact first step, 50 to 100, then keeps showing the model that one house, so its later steps differ from the ones here.

For one weight the whole thing looks silly, since you could have divided $300,000 by 1,000. The point is that the loop doesn’t care how many weights there are, or whether anyone knows a formula for them. Nobody knows a formula for the 8 billion numbers inside an 8B model. Guess, compare, nudge, repeat is how they get set.

The answer key is already in the text

For a language model, what plays the part of the sale price? What is the “right answer” each guess gets compared with?

Here’s the trick. The task is to predict the next token in real text (tokens are the chunks of text a model reads). Show the model the start of a sentence, ask for the next token, and compare its guess with the token that actually comes next. The text grades itself. Then slide along one token and do it again.

Step through it. One puzzle before the last step: a transformer reads the whole sentence at once, so while it’s guessing what follows “the”, the answer “mat” is sitting right there in its input. How can it grade every position in one pass without peeking?

Every position in every document is a training example, and nobody had to write a label for any of them. This is called self-supervised learning. It is supervised in the sense that every guess is checked against a right answer, but the answers come from the data itself. So any large pile of text becomes training data: web pages, books, code, forums, papers. Fifteen trillion tokens of text is about fifteen trillion graded guesses, with no annotators anywhere. The puzzle’s answer is the causal mask from Part 4: each position sees only its own token and the ones before it, never the token it has to predict, so one pass grades every position honestly at once. That’s what makes so many guesses affordable.

This first, largest stage of training is called pre-trainingPre-trainingLarge-scale self-supervised training on next-token prediction.. The “pre” is because more training comes after it, which we’ll get to.

How wrong is “probably mat”?

The house model’s guess was a number, so measuring the miss was easy. A language model’s guess is a probability for every token in its vocabulary (Part 1). After “The cat sat on the” it might say mat 40%, floor 20%, sofa 10%, and a sliver each for tens of thousands of other tokens. The real next token turns out to be mat. How wrong was that?

Pre-training’s answer looks only at the probability p that the model gave to the token that really came next, and charges −log p:

Read it with the two numbers from my notes. If the real token got p = 0.9, the loss is −log 0.9 ≈ 0.105: nearly free. If it got p = 0.01, the loss is −log 0.01 ≈ 4.6. A certain, correct guess (p = 1) costs exactly 0, and as p heads toward 0 the loss heads toward infinity. This is called cross-entropy lossCross-entropy loss−log p(correct token). The standard loss for next-token prediction..

It can look as if this ignores the 60% the model spent on wrong tokens. It doesn’t: the probabilities add up to 1, so raising p(mat) means taking probability away from floor, sofa and everything else, and the nudge pushes each wrong token down in proportion to what it was given.

Why a logarithm? Because a model’s probability for a whole text is the product of its probabilities for each token given the ones before it, and the log turns that product into a sum: −log(p₁ × p₂ × p₃) = −log p₁ − log p₂ − log p₃. So driving down the average loss per token is exactly the same as making the training text, all of it, as probable as possible under the model. There’s also a handy way to read an average loss: e to that power, called perplexity, is how many equally likely options the model is effectively choosing among at each step. A model averaging 4.6 is as unsure as picking one of 100 every time; one averaging 0.105 (perplexity 1.1) is close to certain. Model quality is often quoted this way, and lower is better. For a single guess, e^loss = 1/p only says how surprised the model was: a confident wrong guess can cost 4.6 too.

You have the scoring rule, so now you get to be scored by it.

Text 5 was the trap: “love” wasn’t a candidate. Put all 20 chips on the four and you called it impossible, so your average is now infinite. Even 4 leftover chips, spread over 50,000 tokens, cost 12.4 there, worse than an even spread.

Going all-in pays off only when you’re right. A certain hit costs 0; a certain miss costs infinity, and one infinity ruins any average. (A real model’s softmax never gives a token exactly 0, but 10⁻⁹ still costs about 21, so the lesson stands.) Spreading chips costs a little every time but caps the damage. Somewhere in between is the best strategy, and it has a precise answer: bet your honest odds. If range( is followed by len 40% of the time in real code, put 40% on len. Over many guesses, no other way of betting gets a lower average loss. (Statisticians call a scoring rule with this property proper.) That’s why a freshly pre-trained model’s probabilities mean something: the training pushed them toward how often things really follow each other in text. (The later tuning of Part 8 is known to blur this somewhat.)

Two more numbers for scale. A model that has learned nothing spreads its bets evenly, and over 50,000 tokens that costs ln 50,000 ≈ 10.8 on every guess, roughly where a freshly initialized model starts. At the other end, the range( text showed a floor: where the text itself is uncertain, even a perfect model pays something. Training can’t drive the average to zero, only toward the unpredictability of language itself.

And this is the text version of the house example. After “The cat sat on the”, if the model gave mat a low probability, the loss is large, and every weight gets nudged in whichever direction makes mat a little more likely in contexts like this one. Not one weight but billions, each moved a tiny amount after every batch: a large handful of positions whose nudges are averaged together, so no single sentence yanks the weights around.

Watch a tiny model learn

All the pieces are now on the table: text, next-token labels, cross-entropy, nudges. Here they are working together, for real, in your browser. The model below is a small neural network that predicts the next character (characters instead of tokens keep it tiny) from the few characters before it. Its training text is a stripped-down copy of Part 1 of this series: lowercase letters, spaces and full stops only. Its weights start out random. It isn’t a transformer: it looks at a fixed window of characters through one hidden layer, but it learns by exactly the same loss and nudges.

At step 0 the model writes gibberish and the loss is 3.33, which is −log(1/28): an even spread over 28 characters. Within a few hundred steps the gibberish breaks into word-sized chunks separated by spaces, and soon real words float up: the, and, model, token. Nobody told it what a word is. Spaces and spellings are patterns that make the next character easier to predict, so the nudges found them.

The held-out line is the one that matters. Those sentences never contributed a single nudge, yet their loss falls too, from 3.33 to about 1.9. Not as far as the training loss, about 1.5: even the small model does a bit better on text it has practiced on, and that gap is the part that didn’t carry over. The model hasn’t stored the training text; it has picked up regularities of English (and of my writing) that carry over to sentences it has never seen. That’s generalization, and it’s the entire point. A model that only worked on its training text would be a very expensive copy of it.

Now the large model. Its training loss dives far below the small model’s, toward 0.6. But its held-out loss bottoms out early, around 2.3, and then climbs, all the way past random guessing by the end. Both models read the same text about 17 times over. The large one differs in two ways, and in a text this short both turn into traps. It has seven times the weights: 32,364, more numbers than there are characters to learn from (11,576). And it looks 8 characters back instead of 4. At three positions out of four in this training text, the 8 characters leading up to it appear nowhere else in the text, so each works like a lookup key for the character that follows. Once the general patterns are in, the cheapest way to keep lowering the training loss is to store those one-off continuations, the quirks of these exact sentences. That’s memorization. On new text it’s worse than useless: it’s confidently wrong, which cross-entropy punishes hardest. Look at the highlighted stretches in its samples: chunks copied word for word from its training text.

Why guessing the next word teaches so much

Something here should feel strange. The loss only ever asks about the next token. So how does a next-token predictor end up knowing facts, writing code and following an argument?

Because predicting well requires them. Go back to the blanks at the top of this post: one needed a fact, one arithmetic, one Python, one an idiom and a sense of mood. A model with those skills pays less loss, at billions of positions, than a model without them, so the nudges keep pushing toward them. Patterns that are everywhere, like grammar, get learned early because they pay off at nearly every position. You watched this in Fig. 4: spaces and short words came first. Facts, coding habits and longer chains of reasoning pay off less often, so they need more data and more weights before they’re worth storing. The skills transfer to inputs the model has never seen, because the same grammar, the same facts and the same habits of argument recur across billions of different sentences.

Thousands of GPUs, and a law

Fig. 4 carried a warning: a bigger model isn’t automatically better, because more room needs more text to fill it. Real runs take that seriously. Our tiny model read 192,000 characters in a minute or so; a real run reads trillions of tokens through billions of weights. How much work is that?

A useful rule of thumb: training costs about 6 floating-point operations (FLOPs) per weight per token. Two of them are the forward pass, the same 2 × N per token as inference in Part 3, and about four are the backward pass that works out the nudges. For a model the size of Llama 3 8B, trained on 15 trillion tokens:

That’s about three weeks on 1,000 GPUs, and it’s a floor. It assumes every GPU does useful math at 40% of its peak the whole time (about as good as big runs get), and it leaves out attention’s own arithmetic, GPUs waiting on each other, and restarts after failures. Meta reported about 1.3 million GPU-hours for Llama 3 8B, roughly two and a half times this estimate. The biggest runs go much further: Llama 3.1 405B took about 3.8 × 10²⁵ FLOPs, some fifty times more, on up to 16,000 GPUs for months. That’s what “large” costs.

Why would anyone pay it? Because the payoff is predictable. When researchers trained many models of different sizes on different amounts of data, the loss didn’t improve in fits and starts. It fell smoothly and predictably as parameters, data and compute grew together, so closely that you can train a series of small models, fit a curve, and forecast the loss of a much bigger run before paying for it. These curves are called scaling laws.

Both, about equally. In this picture every tenfold increase in compute buys a model about 3.2 times bigger, trained on about 3.2 times more tokens (3.2 × 3.2 ≈ 10), and takes roughly a third off the part of the loss above the floor of 1.7: at the starting budget, 0.16 above it becomes 0.11. At a single 10× step the lopsided choices are only a little worse (1.816 against 1.807), because near the best split the curve is flat. But the part you don’t grow becomes a wall: put a thousand times the compute all into model size and the loss stalls near 1.79, while growing both reaches 1.75.

Each grey curve is one fixed model size: give it more tokens and it improves, then flattens, because a small model runs out of room to store more patterns. Go the other way, a model too big for its budget, and the budget pays for only a few tokens per weight, so training stops long before those weights are put to use. That isn’t Fig. 4’s memorization: a real run reads most of its text about once, so the oversized model is simply under-fed. The lowest loss for a given budget comes from growing both together, which is the thick curve.

What comes out: a base model

At the end of pre-training you have a base modelBase modelModel after pre-training only. Good at continuing text, not yet an assistant.: an extraordinary text continuer. Give it the start of a document and it writes a plausible rest of that document. That isn’t the same as an assistant. Type “How do I reverse a list in Python?” and a base model might answer, or it might carry on the way such a line often continues on a forum page: with more questions, a username, a list of related threads. It continues text; it doesn’t reliably follow instructions.

Turning it into an assistant is the job of fine-tuningFine-tuningFurther training of a pre-trained model on a smaller, curated dataset. (Part 7), which reuses this same loss on a far smaller, carefully chosen set of examples of good answers, and reinforcement learning (Part 8), which nudges the model with a score instead of a right answer. Those stages shape how the model behaves. Most of what it knows was laid down here.

Three things pre-training bakes in

It stops at its cutoff

The model only knows what was in its training data, and that data was collected up to some date: its knowledge cutoff. Ask about anything later and the model has nothing to go on, but it still produces a fluent continuation, because fluent continuation is all it was ever trained to do. Its weights don’t change when you use it (Part 3), so short of training it again, the only way to get newer facts in is through the prompt.

It can recite

Text that appears many times in the training data can be memorized word for word: license texts, famous quotations, popular code snippets, and sometimes things that should never have been there, like personal details that were posted in many places. Fig. 4 showed the ingredients: text seen over and over, and enough capacity to store it. A real LLM has billions of weights, plenty of room for a string it meets thousands of times, so repetition is usually what decides (though bigger models memorize more of what they see). Pre-training sees most documents only once or a few times, which pushes toward general patterns, but a string duplicated thousands of times across the web gets the treatment the toy gave its whole training text. It’s one reason labs deduplicate their training data, and a reason to think hard about what goes into it at all.

It learns the bad with the good

This is the Honest note above, seen from another side: biases and errors in the data are learned along with everything else, by the same mechanism that learns grammar. The loss rewards predicting text as it was written. If some group is described in stereotyped ways more often than not, the model’s probabilities reflect that. Languages and viewpoints that are rare in the data get less of the model’s ability. Filtering the data helps, and so does the later training in Parts 7 and 8, but pre-training is where these patterns get in.

One question has been hiding under every “nudge” in this post. With billions of weights, how does training know which way to turn each one, and by how much? That’s backpropagation, and it’s next.