Take a language model straight out of pre-training and type a simple question: What is the capital of France? You’d expect Paris. Often you get something more like this:
What is the capital of Germany?
What is the capital of Italy?
What is the capital ofThe model isn’t broken, and it isn’t ignorant. It has read Paris next to capital of France more times than anyone could count. It’s doing exactly what it was trained to do: continue the document. And on the web, a line like What is the capital of France? usually sits in a quiz, a worksheet or a stack of flashcards, where the next line is another question.
Every chat assistant you’ve used started as a model like this. What turned it into something that answers is more training, starting with a stage called fine-tuning, and the surprising part is how little that stage changes. Same model, same loss, same learning algorithm. Only the text is different.
A model that knows the answer but won’t say it
A model fresh out of pre-training is called a base modelBase modelModel after pre-training only. Good at continuing text, not yet an assistant.. Part 5 built one: show it trillions of tokens of text, ask it at every position to guess the next token, and nudge its weights toward the real one. What comes out continues documents. Send it Summarize this email: and an email, and it writes a second email. Give it half a Python function and it finishes the function. Give it a question and it continues whatever kind of document that question usually appears in.
Paris is there, in fourth place at 6% (the figure’s numbers are illustrative, but the pattern is the real one), behind What (another question), A (as in a multiple-choice list) and The. Sample a few replies and you get quiz lists, flashcards and a forum post, all perfectly reasonable ways for a document to go on. The model knows the answer. Answering just isn’t what usually comes next.
The middle setting proves the knowledge is in there. Wrap the question in Q: and A: and the same base model, with the same weights, puts Paris first at 71%, because text shaped like an answer key is usually followed by the answer. Before fine-tuned assistants existed, this is how people got useful work out of base models: they wrote the beginning of a document whose natural continuation was the thing they wanted. But look at what it does next. It answers, then writes the next question, and the next answer, until the length limit cuts it off. Nothing in an answer key says “stop here”.
The fine-tuned model answers Paris. and stops. Now look at the examples it was fine-tuned on, at the bottom of the figure: none of them is about France. What they taught it is what to do when a user asks something: answer, then end the turn.
Same training loop, different text
How do you change a habit? You show the model thousands of examples of the habit you want, and keep training. That’s the whole idea of fine-tuningFine-tuningFurther training of a pre-trained model on a smaller, curated dataset.: take a pre-trained model and continue training it on a much smaller, carefully curated dataset aimed at a specific behavior or task.
Look at what carries over from pre-training. The model still reads a sequence of tokens and predicts each next one. The loss is still the cross-entropyCross-entropy loss−log p(correct token). The standard loss for next-token prediction. from Part 5: −ln p (Part 5 wrote −log p; both mean the natural log), the model’s surprise at the token that actually came next. The weights are still nudged by backpropagationBackpropagationAlgorithm that computes how much each weight contributed to the error, using the chain rule, working from output to input. and gradient steps (Part 6).
You can read the loss straight off Fig. 1. The base model gives Paris a probability of 0.06 as the first word, so if the reply should start with Paris, its surprise is −ln 0.06 = 2.81. The fine-tuned model gives it 0.62, and −ln 0.62 = 0.48. Fine-tuning measured that same surprise on thousands of other questions and their ideal answers, and each gradient step nudged the weights to lower it on a batch of those examples. Because what the model learned was a habit rather than a fact, the surprise at Paris fell too, on a question it never trained on.
The data differs in two ways: what it is, and how much of it there is. Pre-training text is whatever was out there: web pages, books, code. Fine-tuning data is written on purpose, as pairs of (prompt, ideal response): a question and a helpful answer, a support ticket and the reply your best agent would write. Training on such pairs is called supervised fine-tuning, or SFTSFTSupervised fine-tuning on (prompt, ideal response) pairs.. When the pairs cover a broad mix of instructions (summarize this, draft that, explain the other) so that the model learns to follow instructions in general, it’s also called instruction tuning. The pairs are written by people, or generated by a stronger model and then filtered, by people or by automatic checks.
And there’s far less of it. A fine-tuning set of 5,000 examples at about 500 tokens each is 2.5 million tokens. Llama 3 was pre-trained on over 15 trillion tokens, six million times more. Fine-tuning is also gentler. Llama 2’s chat model, for example, was fine-tuned for 2 passes over about 27,500 carefully written examples, at a learning rateLearning rateStep size used when updating weights from gradients. of 2×10⁻⁵, roughly ten times below the peak it was pre-trained with. The model gets adjusted, not rebuilt.
What the model actually reads
The clean picture of (prompt, response) pairs hides one practical detail. A model doesn’t see pairs. It sees one long sequence of tokens. So every conversation is flattened into text with a fixed layout, called a chat template. Special tokens, which exist only for this job, mark where each turn starts and ends and who is speaking: the system (the app’s standing instructions), the user, or the assistant. The template below is in the ChatML style, with <|im_start|> and <|im_end|>. Llama 3 uses <|start_header_id|> and <|eot_id|>, Gemma uses <start_of_turn>, Mistral uses [INST] … [/INST] (plain text in its first versions, special tokens in later ones). Every model family has its own, and a model is only reliable with the one it was trained with.
Three tokens: Paris, the full stop, and the closing <|im_end|>. Not the <|im_start|>assistant↵ header before them: when the model is used, the app writes that header itself, so the model never has to produce it. With the mask off, every token but the first is graded (the first has nothing before it to predict it from), and the loss is dominated by tokens nobody could predict: the words of the system prompt and of the user’s question. Tap capital or France and you’ll see surprises above 3. Only 8% of the unmasked loss comes from the reply, so training on all of it would mostly teach the model to guess what users are going to ask, which isn’t its job. With the mask on, the loss is the average surprise over the reply alone: (0.80 + 0.48 + 0.34) / 3 = 0.54. These probabilities come from a model partway through fine-tuning, which is why Paris sits at 0.45 here rather than Fig. 1’s 0.62.
Look at the last token in R: <|im_end|>. Grading it is how the model learns to stop. The base model in Fig. 1 kept going because nothing in its training text marked where an answer ends. It has seen plenty of documents end, at the end-of-sequence token from Part 1, and plenty of dialogue, from forum threads to interview transcripts. But an answer in the middle of a document doesn’t end the document, so no token ever said “this turn is over”. The fine-tuned one emits its end-of-turn token, and the app stops generating when it sees it.
Masking the prompt is the usual choice, not a law of nature. Some recipes train on prompt tokens too, or give them a smaller weight. Either way it’s a decision about which tokens count, made while preparing the data. The training loop doesn’t change.
The bottom half of the figure shows why the template keeps mattering after training. Before you switch it to Hand-written, guess: same words, but plain User: and Assistant: lines instead of special tokens. Will the model still stop after Paris.? Often it won’t. To this model, the hand-written prompt is no longer a conversation with a slot for its reply. It’s a document that looks like a transcript, and the natural continuation of a transcript is more transcript: an answer, then your next line, written for you, then an answer to that.
Jobs where the habit is the problem
If fine-tuning changes habits, its good uses are the jobs where a habit is the problem. A consistent format, like “always reply with valid JSON in this schema”, is easy to say and hard to guarantee with a prompt alone; a few hundred examples in exactly that shape make it the default. A tone or domain voice (a brand’s way of writing, a clinician’s register, a terse code reviewer) is hard to describe and easy to show. Narrow classification or extraction, like sorting tickets into twelve categories or pulling the parties and dates out of a contract, is a job a fine-tuned model, often a much smaller one, does quickly, cheaply and far more consistently than a prompted general model.
The fourth is distillation. Run a large, expensive model (the teacher) on thousands of real prompts, keep its good answers, and fine-tune a small model (the student) on those pairs. The student learns to imitate the teacher on that job and costs a fraction as much to run. If you can get the teacher’s full probability distributions, the student can learn to match those instead of only the tokens the teacher picked, which carries more information per example.
Fine-tune or retrieve?
The tempting use is a different one: “Our model doesn’t know our product catalog, so let’s fine-tune it on the catalog.” For facts that change, that’s a poor and expensive choice, and it’s worth seeing why.
Facts learned in fine-tuning live in the weights, so every change means a new training run, with its cost and its testing, and between runs the model is confidently out of date. Facts also don’t stick from a single mention: a model needs to see one many times, in different phrasings, before it recalls it reliably, and even then it may blend the new price with the old one. It can’t tell you where an answer came from. And you can’t easily take a fact back out.
The alternative leaves the weights alone and puts the facts in the prompt: look up the relevant text when the question arrives and paste it in. That’s retrieval-augmented generation, RAGRAGRetrieval-Augmented Generation. Retrieve relevant text and add it to the prompt before generating., the subject of Part 12. Updating a fact means editing a document. Hence the rule of thumb: fine-tune for behavior, format and style; retrieve for knowledge, especially knowledge that changes. Try it on seven requests a team might bring you. There’s also a Just prompt button, because sometimes neither is needed. Commit to an answer before you click.
Many real systems end up looking like the last request: tuned for how to behave, with retrieval or tools for what’s true today. And as the first request showed, try a good prompt with a few examples in it before either. It costs nothing to change and often gets you most of the way. Fine-tuning earns its cost when the prompt has been tried and the habit still won’t hold, or when a small tuned model can replace an expensive prompted one.
Where fine-tuning bites
Fine-tuning is easy to start and easy to get wrong. The four classic ways all come from the same fact: the model learns whatever the data and the loss reward, faithfully, and nothing else. One of them, the wrong chat template, you’ve already met. Here are the other three.
Bad examples teach bad habits
Take a job fine-tuning is good at: turn a customer’s message into JSON with two fields. You’ve scraped 5,000 example pairs from old tickets, and below is a sample of ten. Each kind of flaw in it shows up in exactly one of the ten, so the model will learn each as a 10% habit.
Four in ten. Each flaw on its own is rare, but no example has two of them, so a model that matches its data makes each flaw 10% of the time and turns out a clean output only 60% of the time: four 10% habits add up to 40%. A real model fits its data more loosely and can even mix habits into combinations no example had, like a chatty preface and single quotes. Drop the four bad cards and every habit falls to zero: 3,000 rows instead of 5,000, and a better model.
Now consider what more data would have done. Scrape 50,000 rows with the same flaws in the same proportions and each habit stays at 10%; the model just learns them more precisely. Volume doesn’t average out an inconsistency, because the loss rewards reproducing it. That’s the reasoning behind a rule practitioners repeat: a few hundred to a few thousand excellent examples often beat a large noisy set.
Learn B, lose A
The next figure trains a real neural network in your browser: 337 weights, small enough to watch. It has already learned task A, sorting points on the left half of the square by which side of a wave they fall. That’s its “general skill”. Now fine-tune it on task B, a different pattern on the right half. Task B’s data says nothing at all about the left half. So what should happen to task A while the network trains on B alone?
Task A collapses, from 99% to 52%, about as good as a coin flip, while task B climbs to 97%. Nothing in task B contradicted task A. The trouble is that the loss only measured B. Every weight in this network serves both tasks. Backpropagation finds, for every weight, the direction that helps B, and the optimizer moves it that way, with no term in the loss saying “and don’t break A”. The left half of the map gets redrawn as a side effect. This is called catastrophic forgetting. In a fine-tuned LLM it shows up as general skills getting worse: a model tuned hard on contract summaries can get clumsier at code, at casual chat, at following other kinds of instructions.
Now set Mix in old data to 10% and run it again. Three examples in every batch of 32 come from task A. A still dips early, to about 62%, but the old data keeps it in the loss, so it climbs back to 94% while B still reaches 97%. Mixing some of the original kind of data back in is one standard defense, and the one this toy shows best. The others work by moving the weights less: a lower learning rate and fewer steps, and methods like LoRALoRALow-Rank Adaptation. Freezes the base weights and trains small low-rank adapter matrices. (Part 9), which freeze the original weights and train a small add-on. These shrink forgetting rather than rule it out, because a smaller change to shared weights is still a change; in this toy, a learning rate three times lower only delays the collapse. Whatever you use, measure the general skills before and after, not only the task you tuned for.
Overfitting a small set
Back to the figure for the last pitfall. Set Mix in old data back to None and switch to 16 examples. Before you run it, guess: will it score better or worse on its own training points than the 300-example run did? It reaches 100% on its 16 training points, better than before, but only 78% on new task-B points, down from 97%. In 600 steps of 32 draws it saw each of the 16 points about 1,200 times: plenty of chances to fit each one without finding the rule. Watch its solid boundary bend around the squares instead of following the dashed lines. That’s overfitting, and small fine-tuning sets invite it, especially with many passes over the same examples. In an LLM it looks like a model that parrots its training replies, phrasing and all, and handles anything unlike them badly. The defenses are Part 6’s: more varied examples, a validation set you never train on, and early stopping, which keeps the checkpoint where validation loss was lowest rather than the last one.
Everything here needed someone to write the ideal answer. For much of what we want from a model, nobody can write one ideal answer, but people can easily say which of two answers is better. Learning from that kind of signal, a score instead of an answer, is next.