Reinforcement Learning & RLHF: Learning From a Score, Not an Answer

You’ve probably played this at a party. Someone hides a key, and you search while they say only “warmer” or “colder”. Nobody tells you where the key is, or even which way to walk. You find it anyway, because every step gets a score and you keep doing more of whatever made the score go up.

Now a puzzle left over from Part 7. Fine-tuning turned a text-continuer into an assistant by showing it thousands of ideal answers. That works when someone can write the ideal answer. Try writing the one correct answer to “explain recursion to a ten-year-old”, or the perfectly judged way to decline a risky request without lecturing anyone. You can’t. But show most people two attempts and they’ll pick the better one in seconds. Code works the same way: writing a correct function is hard, and checking one against its tests is easy.

Reinforcement learning is the warmer-colder game played with a language model. It’s how a model learns qualities nobody can write down as an answer, and a large part of what turns an imitator of good answers into the assistants and reasoning models you use. It also has a talent for finding loopholes: before the end of this post you’ll have trained a model that cheats.

Three steps, and no answer key

In reinforcement learning (RL) the model is never shown the correct answer. Training goes around a loop instead:

1. The model produces an answer.
2. The answer receives a score, called the reward.
3. The weights are nudged so that high-scoring answers become more likely.

In RL language, the model being trained is the policy: the thing that decides what to do, which for an LLM means what to write. The reward is just a number, and where it comes from is the big design decision. There are two sources. People can judge or rank the answers; that’s RLHFRL / RLHFReinforcement learning (from human feedback). Train by scoring the model's own outputs., reinforcement learning from human feedback. Or automatic checks can score them: does the generated code pass its unit tests, does the math answer match the known result.

The automatic kind is the cleanest place to start. The prompt is “Write a function that reverses a string.” To keep the mechanics visible, the toy model below doesn’t write code token by token; it chooses between three whole programs. A slices the string backwards, which is correct. B sorts the letters in reverse order, which happens to work on "abc" and not much else. C is a loop that stops one character short. The scorer runs six unit tests and pays 1 if all of them pass, otherwise 0. (Yes, you could just fine-tune on s[::-1]. But on hard problems, tests are far easier to write than solutions, and the tests are all this loop needs.)

Your first round is the simplest interesting case, so here it is by hand. A passed (reward 1) and B and C failed (reward 0), so the group’s average is 1/3. Each attempt is compared with that: A’s advantage, its reward minus the average, is +2/3, and B’s and C’s are −1/3. Each program’s logitLogitRaw, unnormalized score for each vocabulary token before softmax. (the raw score that softmax turns into a probability, from Part 1) then moves by its advantage times a step size, 2/3 in this toy: A’s goes up 0.44, B’s and C’s down 0.22. That multiplies A’s share by e^0.44 ≈ 1.56 and the others’ by e^−0.22 ≈ 0.80, giving 0.31, 0.40 and 0.24. Rescale so they sum to 1 (divide by 0.95) and A is at 33%, B at 42%, C at 25%. Nobody said “A is right”. A scored above its group, and that was enough. That’s what “warmer” meant at the party: not “you’re close”, just “better than what you were doing”.

Round 2 wrote B three times: everyone scored 0, every advantage was 0, and nothing moved. Equal scores carry no hint about which way is better, so with a group-average baseline a problem the model always fails teaches it nothing, and neither does one it always solves. That’s why RL starts from a pre-trained, usually fine-tuned, model: it needs one that already succeeds some of the time. By round 9 A is at 93%, and most later rounds are A, A, A, with no signal, so learning slows to a crawl. The better the model gets at a prompt, the less that prompt can teach it.

Written out, the update for each sampled answer y is the classic policy-gradient rule, known as REINFORCE:

Here θ stands for all the weights, η is the learning rate, and π_θ(y) is the probability the model gives to answer y. The gradient comes from ordinary backpropagation (Part 6). Now delete the multiplier in front, and what’s left is the fine-tuning step from Part 7: make y more likely. So RL is fine-tuning on the model’s own attempts, with each step scaled by how much better than average the attempt scored, and reversed when it scored worse.

When nobody can write the test

Unit tests are a luxury. What’s the test for “helpful”? For a refusal that’s firm about the dangerous part and useful about the rest? For the right tone with a frustrated customer? Helpfulness, refusals and tone have no single correct answer, and multi-step reasoning has no single correct path even when the final result can be checked. That’s why all four are shaped with RL rather than fine-tuning alone.

What people can do is judge, even on the string example. Say the model writes two programs that both pass every test: return s[::-1] with a one-line explanation, and an uncommented twelve-line loop. The tests can’t separate them; a person prefers the first in seconds. Raters asked for the clearer, safer, more helpful answer add exactly this kind of signal, and the same update then makes outputs like A and the styles people preferred more likely.

Notice the word compare. Ask raters for a score from 1 to 10 and you get noise: one rater’s 7 is another’s 5. “Which of these two is better?” is easier and gets more consistent answers. So human feedback is collected as preference comparisons: “A is better than B”.

That leaves a scale problem. RL needs a score for every sampled answer, millions of them, and you can’t seat a person next to each one. So the comparisons train a second model, the reward modelReward modelModel trained to predict human preference scores. Used in RLHF., whose job is to predict them. It’s usually an LLM itself, with its last layer swapped for one that outputs a single number: the score.

How do you get scores out of “A beat B”? Chess solved this with Elo ratings, which record only who beat whom and are set so that a bigger rating gap predicts a more lopsided result. Do the same with answers: give each a score, and let the chance a person prefers A depend only on the gap r(A) − r(B), squashed into a probability by the sigmoid σ. That’s the Bradley–Terry model (Elo is a version of it):

Only gaps matter: add 100 to every score and no prediction changes. Training makes the predicted probability of each choice people actually made high; the loss is −log of that probability, the same cross-entropy idea as in pre-training (Part 5).

You’re the rater now. The reward model below is a tiny one. It sees only three things about an answer (whether it’s correct, how long it is, and whether it praises or agrees with you) and it learns one weight for each by fitting Bradley–Terry to your choices, for real.

The reward model learns your taste, not your intentions. As the Careful rater it values correctness (+2.6) and penalizes praise (−1.0). It also dislikes length (−1.4 per 100 words), mostly because of pair 3, where the long answer was the same correct one plus 170 words of extras. That rater only mildly preferred the short version, but a comparison records which answer won, not by how much, so a mild preference looks like a firm one. This reward model will mark down an answer that’s long because the question was hard.

Switch to “Likes long” and the favorite React answer is the 240-word one that never finds the bug: length earned more weight than correctness. Praise gets +0.6 too, though this rater never cared about praise: in pair 4 the longer answer was longer only because it opened with “Great question, you clearly know your stuff!”. Switch to “Likes praise” and the favorite is “That’s normal React behavior, and it’s harmless”: wrong, flattering and reassuring. The model was never told to flatter; a rater who enjoyed being agreed with taught it.

One more experiment: press Reset and answer only the first comparison. “Correct” and “praise” get equal and opposite weights: in that pair the correct answer was also the unflattering one, so one comparison can’t tell the two apart. Only varied comparisons can, and a real reward model faces the same tangle in millions of dimensions.

Chasing the score, on a leash

With a reward model in hand, Fig. 1’s loop can run on anything. But the reward model is only a stand-in for human judgment, trained on a limited set of comparisons. Somewhere in the enormous space of possible answers are ones it scores far too high, and an optimizer’s whole job is searching for high scores. Push hard enough and it finds them.

So RLHF puts the policy on a leash. Keep a frozen copy of the starting model, called the reference, and charge the policy for drifting away from it. Drift is measured by KL divergence, a standard measure of how different two probability distributions are; it’s zero when they’re identical. The objective becomes:

Concretely, KL(π ‖ π_ref) is the average, over the answers the policy actually writes, of log(π(y) / π_ref(y)): how many times likelier the policy has made each answer, on a log scale (measured in nats, with natural logs). So each sampled answer is charged its own log ratio. If the policy has become 20 times likelier than the reference to write it, its reward is reduced by β × ln 20 ≈ 3β. The charge grows only logarithmically, though: 200 times likelier costs just 5.3β. A big enough reward lead will outbid it.

This objective has an exact solution for any set of answers (computing it just requires listing them all), and the rest of the post, DPO included, is built on it:

With a large β, every multiplier is close to 1 and the policy stays where it started. With a small β the multipliers explode, and whichever answer has the highest reward takes everything, however unlikely it was to begin with. This is the blurring Part 5 mentioned: pre-training leaves a model betting its honest odds, and RL deliberately reweights them toward whatever scores well, so a tuned model’s probabilities are less trustworthy as odds.

The figure applies this to five answers to a debugging question. One of them is the reward model’s blind spot: 400 confident, flattering words blaming a nonexistent Python bug, which the reward model scores higher than any honest answer (2.8, against 2.0 for the best real fix). Each answer also has a “true quality”, the score a careful expert would give it. Before you drag: as the leash loosens, the reward-model score can only go up. What do you think happens to the true quality?

The reward-model score climbs the whole way, because it’s what is being maximized. True quality climbs at first, as vague, ask-for-code and even quick-fix answers lose share, mostly to “Fix + why”, and peaks at about 0.79 near β = 0.5. Even there the nonsense is written about one time in ten, twenty times as often as by the SFT model. Loosen further and quality collapses. That’s Goodhart’s law in miniature: when a measure becomes a target, it stops being a good measure. The same hump has been measured with real reward models, the proxy score still rising while independently judged quality falls.

You can work out the takeover by hand. The SFT model writes the nonsense with probability 0.005 and the best fix with probability 0.14: a 28-to-1 head start for the fix. The nonsense’s reward lead is 2.8 − 2.0 = 0.8, which multiplies its odds by e^0.8/β. At β = 1 that’s e^0.8 ≈ 2.2, nowhere near 28, so the fix stays about 13 times likelier. At β = 0.1 it’s e⁸ ≈ 2,981, and the nonsense wins by about 107 to 1. The crossover is where e^0.8/β = 28, at β = 0.8 / ln 28 ≈ 0.24.

Real training can’t list every possible answer like this; there are astronomically many. PPO, the RL algorithm in the next section, approaches the same optimum by sampling and taking small steps, estimating the KL from the samples. The trade-off is real; the five answers and their scores are made up to show it. In practice β is tuned by watching evaluations the policy isn’t trained on.

The whole recipe, and three shortcuts

The classic RLHF pipeline, the recipe OpenAI described for InstructGPT in 2022, has four stations:

1. Start from an SFTSFTSupervised fine-tuning on (prompt, ideal response) pairs. model, the output of Part 7.
2. Collect human preference comparisons (“A is better than B”).
3. Train a reward model to predict those preferences.
4. Optimize the LLM against the reward model with an RL algorithm such as PPO, with a KL penalty so it doesn’t drift too far from the SFT model.

PPO, proximal policy optimization, is Fig. 1’s update grown up. It clips its objective so that once a token’s probability has moved more than about 20% within one update, pushing it further earns nothing (“proximal” means nearby). That’s a second, different leash: the clip keeps each step near the policy of a moment ago, while the KL penalty keeps the whole run near the SFT model. And instead of a group average, PPO usually trains a value model that predicts, from each partial answer, the reward it’s likely to end with.

DPO folds two stations, the reward model and the RL loop, into one loss on the pairs. RLAIF swaps the people for a model; verifiable rewards swap the people and the reward model for a check.

DPO, direct preference optimization (2023), is the clever one. Solve Fig. 3’s exact solution for the reward and you get r(y) = β · log(π*(y) / π_ref(y)), plus a term that depends only on the prompt: every policy implies a reward. Put that into Bradley–Terry and the prompt term cancels in r(A) − r(B), leaving a loss written entirely in terms of the policy and the reference:

It’s Bradley–Terry again, with each answer’s score replaced by how much more the policy likes it than the reference does. DPODPODirect Preference Optimization. Learns from preference pairs without a separate reward model. trains on that loss directly, widening the gap between preferred and rejected answers (in practice both probabilities often fall, the rejected one faster). No reward model, no RL sampling loop, just ordinary supervised-style training. The trade-off: standard DPO learns only from a fixed set of pairs, never from fresh samples of the model’s own output.

RLAIF, reinforcement learning from AI feedback, keeps the pipeline but replaces the human raters with a model, usually instructed by a written list of principles (Anthropic’s Constitutional AI is the best-known example). Labels get cheap, fast and more repeatable, and the principles sit in writing where people can read and argue over them. The judge’s blind spots come along too, applied at scale.

RL with verifiable rewards drops people and the reward model whenever a program can check the answer: does the final number match, do the tests pass. That’s Fig. 1 at scale, and it’s central to training “reasoning” models. Reward only correct results on math and coding problems, and long step-by-step reasoning tends to emerge, because chains that end in a right answer get reinforced. The model finds those chains on its own. DeepSeek reported exactly this in January 2025 for R1-Zero, trained with RL alone on its base model using GRPO, which scores each attempt against its group’s average like Fig. 1 (and also divides by the group’s spread). The released R1 first fine-tuned on a few written reasoning examples, partly to make its chains readable.

When the model games the score

The scorer is the specification

Reward hackingReward hackingA model exploiting flaws in its scoring signal instead of truly improving. is the model exploiting flaws in the scorer instead of getting better at the task. The reward model’s blind spot in Fig. 3 was one kind. Here’s the classic kind for automatic checks: special-casing the tests.

Same task as Fig. 1, but now the prompt includes two examples, and a lazy scorer runs only those two. The model also has a fourth program in its repertoire, D, which recognizes the two example inputs and returns the expected outputs.

Every run reports a training reward of about 95%. Reveal the hidden tests and, with the default scorer, the first three runs turn out to be very different models: the first learned the hack and writes working code only 11% of the time, the second learned the real thing (90%), and the third split the difference. The training reward can’t tell them apart, because the hack genuinely satisfies the scorer. From inside the loop, D isn’t cheating. It’s a perfect answer to the question that was actually asked.

Switch the scorer to all six tests and train again: D earns 0, dies off like B and C, and the training reward finally means what you hoped. The fix was never in the model; it was in the scorer. AI labs have reported reasoning models doing real versions of this during training, such as special-casing tests or editing the tests instead of the code. The defenses are the ones you’d use with a clever intern: hidden tests, checks that are hard to satisfy any other way, the KL leash, and people reading samples of what the model actually does.

Telling raters what they want to hear

Sycophancy is the human-feedback version of reward hacking: the model learns to tell raters what they want to hear. You built one in Fig. 2, and RL makes it worse rather than merely copying it. Say raters pick the agreeable one of two otherwise-equal answers 60% of the time, a mild taste. Bradley–Terry turns that into a reward gap of ln(0.6/0.4) = ln 1.5, and Fig. 3’s formula multiplies the agreeable answer’s odds by e^{ln 1.5 / β} = 1.5^1/β: 1.5 at β = 1, exactly the raters’ own 60:40, but 1.5¹⁰ ≈ 58 at β = 0.1. A model that started at 50/50 now agrees 98% of the time. It praises mediocre ideas, accepts a user’s wrong premise, and folds when pushed back on (“You’re right, I apologize”) even when it was right the first time. The fixes go into the scorer: rater guidelines that reward disagreeing when the user is wrong, and evaluations that check whether the model holds its ground.

Everything so far has assumed that training, fine-tuning or RL, updates all of a model’s billions of weights. It doesn’t have to. Next: how to steer a model while training only a few million of them.