Backpropagation: Which Way Should Each Knob Turn?

Tune a guitar string by ear and you run a tiny loop: pluck, hear that it’s flat, tighten the peg a little, pluck again. Your ear hands you two things for free: which way to turn, and roughly how far off you are.

Now picture a guitar with eight billion pegs. Every peg changes the sound of every string, and all you get to hear is one chord and a single number saying how wrong it sounds. Which pegs do you turn? Which way? By how much?

That is a language model in training. It reads some text, guesses the next token, and is told how wrong the guess was (Part 5). The guess came out of billions of weightsWeightsThe learned numbers that scale inputs inside the network. Set during training, frozen at inference. acting together, each one a knob like a guitar peg, and every one of them needs its own instruction: turn this way, about this much. The obvious way to get those instructions is to turn one knob, play the chord again and listen. That’s eight billion test runs for a single round of training. Real training gets every instruction for the price of about two or three runs, using backpropagation, which you’ll run by hand below.

One knob: which way is downhill?

Start where Part 5’s Fig. 1 started: the one-weight house-price model from Part 3, price = w · x, where x is the floor area and w is dollars per square foot. One house: x = 1,000 sq ft, sold for y = $300,000. Training starts from a bad guess, w = 50, so the model predicts $50,000, and this time it sees only that one house, again and again. Part 5 nudged w from 50 to 100 and promised the rule behind the nudge. Here it is, in three moves.

Move one: measure how wrong. For a model that predicts a number, the usual choice is the squared error:

The error is 50,000 − 300,000 = −250,000, and the loss is its square, 6.25 × 10¹⁰. Squaring makes every miss count as positive, punishes big misses far more than small ones, and gives a smooth curve with a slope everywhere, which is exactly what the next move needs. (A language model uses a different loss, cross-entropy, from Part 5. Everything below works the same for any loss that has a slope.)

Move two: which way should w turn? With one knob you can simply try it. Nudge w from 50 to 51: the prediction becomes $51,000, the error −249,000, and the loss 6.2001 × 10¹⁰. The loss fell by 4.99 × 10⁸ for one unit of w. That ratio, change in loss per unit change in the weight, is the slope of the loss curve at w = 50. Calculus gives it exactly, no nudging required:

That number is the weight’s gradientGradientThe direction and size of change in the loss with respect to each weight.. Its sign says which way is uphill: negative means raising w lowers the loss, so w should go up. Its size says how steeply the loss changes right here. Far from the answer the curve is steep and the gradient is big; near the answer it flattens toward zero. (The nudge gave 4.99 rather than 5 because a nudge of a whole unit lets the curve bend along the way. Smaller nudges get closer.)

Move three: step. The gradient points uphill, so step the other way, scaled by a small number η (eta) called the learning rateLearning rateStep size used when updating weights from gradients.:

One step takes w from 50 to 100. Compute the slope, step downhill, repeat: that is gradient descent.

Those three moves are one round of training for any model, from this one-knob toy to an LLM: measure the loss, find every weight’s gradient, and step each weight downhill (the rule that takes the step is the optimizer). Backpropagation is move two, done for billions of weights at once.

The second step lands on 140, not 150, then 172, 197.6, 218.1. The steps shrink because the gradient is proportional to the error, and the error shrinks as w improves. Watch the gap to 300 instead: 250, 200, 160, 128, 102.4. Every step closes exactly 20% of whatever gap is left, so w never quite reaches 300. It just gets as close as you like (after 20 steps the gap is under 3).

Where does 20% come from? Since y = 300x, the gradient 2(w·x − y)·x is 2x²(w − 300), so the update is w_new = w + 2ηx²(300 − w). Subtract both sides from 300:

That multiplier explains every preset in the figure. At η = 10⁻⁸ it’s 0.98: two percent a step, a crawl. At 5 × 10⁻⁷ it’s exactly 0, and one jump lands on 300. At 10⁻⁶ it’s −1: each step overshoots by precisely as much as it was off, and w bounces between 50 and 550 forever. Any bigger and the multiplier drops below −1, every overshoot is larger than the last miss, and the loss explodes.

Don’t read the one-jump preset as a recipe. It works only because this loss is a perfect parabola whose curvature we know, and even the scale of η depends on units (squared dollars here, which is why it’s as tiny as 10⁻⁷). A real loss surface curves differently in every direction, shifts as training moves, and isn’t one bowl but many dips, ridges and flat stretches; gradient descent only promises to keep going downhill from wherever it is, though in practice that finds weights that work well. So people find a good η by experiment and change it during training (more on that near the end).

Billions of knobs, and a nudge test that doesn’t scale

The nudge test works for any weight in any network: nudge it, rerun the model, see how the loss moved. The catch is the bill. Each gradient costs one extra forward pass (one run of the model from input to output), so a model with N weights needs N + 1 runs to get them all. For an 8-billion-weight model, that’s the guitar from the start: eight billion runs for a single training step.

Backpropagation gets every gradient for one forward pass plus one backward pass, and the backward pass costs about as much as one or two forward passes, whatever the model’s size (in full training, about two: that’s where 4 of Part 5’s 6 FLOPs per weight per token go). Its trick is the chain rule, which is friendlier than its reputation.

A network is a chain of simple pieces, stacked in layersLayerA stage of the network whose output feeds the next stage.: multiply by a weight, add things up, apply an activation functionActivation functionNon-linear function (ReLU, GELU…) applied to a neuron's weighted sum. Lets networks model non-linear patterns., multiply again, compute the loss. Each piece on its own has an easy local slope, and slopes along a chain multiply. You’ve already done this once without noticing. In the house model the loss depends on the error (L = error², local slope 2 × error = −500,000), and the error depends on w (error = w·x − y, local slope x = 1,000). Multiply them: −500,000 × 1,000 = −5 × 10⁸, the gradient from before. That’s the chain rule. It works like gear ratios, except that a network’s ratios depend on the values flowing through it, so they’re worked out afresh for every example.

So start at the loss and walk backwards. At each piece, take the gradient arriving from above, multiply it by the piece’s local slope, and pass the result back. When you reach a weight, multiply by the value it was multiplied with on the way forward, and that’s its gradient. This backward walk is backpropagation, short for “backward propagation of errors”. With more than one weight, the slope gets a curly d: ∂L/∂w₂ means “how fast L changes as w₂ moves, with every other weight held still”. It’s the same idea as dL/dw above, one weight at a time.

w₂ should go down, even though the output is too low and every other weight wants to go up. Raising w₂ raises h₂, and h₂ drags the output down through v₂ = −1, making a too-low output lower still. Nobody had to reason that out. The multiplication did it: −2 from the loss, times −1 for v₂, times 1 for the ReLU, times 2 for x, gives +4. A positive gradient: turn it down.

Two more things to notice. First, look at where the −2 went. It was computed once, at the loss, and all four gradients are that −2 times a few more local slopes. In a deep network each layer’s running product is computed once and handed down to every weight below it, so no weight pays for its own trip back to the loss. That sharing is why the whole backward pass costs about one or two forward passes, where the nudge test needed a fresh run per weight. Second, the backward pass kept reaching for values from the forward pass: h₁ for v₁’s gradient, x for w₁’s. Training has to keep those values around, and that costs memory (more on that bill shortly).

Backprop computes, the optimizer applies

Backprop’s output is one number per weight, and that is all it does. Deciding what to do with those numbers is the optimizer’s job. The simplest optimizer applies the house-model rule to every weight at once, w ← w − η·g, and in practice it runs on mini-batches. The loss you care about is the average over all the training data, but its exact gradient would need every example run through the model before a single step. So each step takes a batch (for LLMs, often millions of tokens), averages the gradients over it, and steps. The batch gradient is a noisy estimate of the true one: right on average, and vastly cheaper. Gradient descent on random mini-batches is called stochastic gradient descent, or SGD.

Plain SGD has a weakness that shows up as soon as there are two weights. Loss surfaces are rarely round bowls; often they’re long, narrow valleys, steep across and nearly flat along. Fig. 1’s gap law still holds, one direction at a time, with x² replaced by the average square of that direction’s input. In the valley below that’s about 9.4 one way and 0.068 the other (inputs of typically 3 and 0.25), 140 times apart. At the η plain SGD uses there, 0.095, the steep multiplier is 1 − 2 × 0.095 × 9.4 ≈ −0.79: every step overshoots across the valley. The gentle one is 1 − 2 × 0.095 × 0.068 ≈ 0.987: the floor closes 1.3% per step. Raise η past about 0.106 to hurry along the floor and the steep multiplier drops below −1, Fig. 1’s blow-up. One learning rate can’t suit both directions, so the weights zigzag from wall to wall while creeping forward.

Two fixes are standard. Momentum keeps a fading sum of recent gradients (each step keeps 90% of the old total and adds the new gradient) and steps along that. Pushes that keep pointing the same way, like the gentle slope along the floor, build up speed; pushes that flip back and forth partly cancel. It behaves like a heavy ball: faster, but prone to swinging. Adam keeps a momentum-style average and a running average of each weight’s squared gradient, and divides each weight’s step by the square root of the second. Large gradients get their steps scaled down and tiny ones get theirs scaled up: when a weight’s gradient keeps roughly the same sign and size from step to step, m/√s is close to ±1, so every weight moves about η per step, whatever the size of its gradient. In effect, every weight gets its own learning rate.

AdamW, the usual choice for LLMs, is Adam plus weight decay: every step also shrinks the weights slightly toward zero (usually not biases or normalization gains), separately from the gradient. It’s a gentle pressure against large weights, and it comes back later under overfitting.

SGD’s first step goes almost straight across the valley, because that’s where the slope is. It zigzags, settles onto the floor, then crawls: 0.987⁸⁰ ≈ 0.35, so after 80 steps it has covered only about two-thirds of the valley. Momentum swings from side to side like a ball in a gutter, but it builds up speed along the floor and settles by step 55. Adam settles first, at step 46. From its very first step it moves along the valley as much as across it (that step is exactly η in each weight’s direction), because dividing by each weight’s typical gradient size evens out the steep and the gentle directions. (That works best when the steep and gentle directions line up with individual weights, as they do here. Adam isn’t magic for every valley, but in large networks it helps enough to be the default.)

Check the readouts at step 80, though: momentum reads slightly below Adam. Its leftover zigzag makes its loss jump from step to step, and step 80 happens to catch it low (Adam is lower on most steps from 60 to 80). Neither has landed. Both overshot the cross along the floor and curled back (the little loop to its right), carried on by their running averages after the slope flattened. Run them longer and both close in.

With mini-batches on, the paths go ragged and none of them settles exactly on the cross, because each batch has a slightly different idea of where the bottom is. SGD usually suffers most (press New batches a few times): its across-valley multiplier was already −0.79, close to Fig. 1’s bounce point of −1, and a batch whose inputs happen to run large tips it over. Adam, whose steps don’t grow with the gradient, copes best.

Why training needs eight times the memory

Training has to hold much more than the model. The backward pass needs the forward pass’s intermediate values (you saw it reach back for h₁ and x), so training keeps these activations for every layer until they’ve been used. It needs a gradient for every weight. And the optimizer keeps its own state: one extra number per weight for momentum, two for Adam. In the usual mixed-precision recipe, weights and gradients are stored in 16 bits, while the optimizer state and a master copy of the weights are kept in 32 bits, so that tiny updates aren’t rounded away.

So here’s a question to answer before you touch the calculator below: an 8-billion-weight model needs 16 GB just to hold its weights for inference, at 2 bytes each. How much does it need to train with Adam, before counting activations?

128 GB: 16 bytes for every weight instead of 2, eight times what it takes to run, and the activations come on top. They grow with batch size and sequence length, and at long sequences they can rival or exceed everything else on the bill. A common trick called activation checkpointing throws most of them away and recomputes them during the backward pass, trading extra compute for memory. Click 70B, a round 70 billion weights (Llama 3 70B’s actual 70.6 billion make the 141 GB of Parts 3 and 9): 140 GB to run, two 80 GB cards; 1.12 TB to train, fourteen cards before a single activation. That bill is what the cheap-adaptation tricks later in the series attack: LoRA freezes the original weights and trains a small add-on, so gradients and optimizer state are needed only for the add-on’s few million weights.

Three ways training goes wrong

A learning rate that has to change over time

Fig. 1 showed the extremes: too large and training oscillates or diverges, too small and it crawls. Real training adds a twist: the best learning rate changes as training goes. At the start, the weights are random, the gradients are large and erratic, and Adam’s running averages have seen almost nothing, so its per-weight step sizes can’t be trusted yet. One full-size step then can throw the model somewhere it struggles to recover from. So the standard schedule begins with a warm-up, ramping the learning rate up from near zero over the first stretch of training. Afterwards it decays, often along a cosine curve, because late in training the mini-batch jitter from Fig. 3 keeps full-size steps bouncing around the bottom, and smaller steps let the weights settle.

Fifty layers of 0.9: vanishing and exploding gradients

The chain rule says an early weight’s gradient is a product of local slopes, one for every layer between that weight and the loss. A product of many numbers is touchy about their typical size: fifty factors of 0.9 multiply to about 0.005, and fifty factors of 1.1 to about 117. If the local slopes run a little small, the gradient fades to almost nothing by the time it reaches the early layers, and they barely learn. That’s a vanishing gradient. If the slopes run a little large, the gradient grows huge and a single step wrecks the weights: an exploding gradient. Older networks built on the sigmoid activation, whose slope is never more than 0.25, ran straight into the first problem: ten such layers can shrink a gradient a million times.

Three standard defenses:

• Residual connections. Each block adds its output to its input instead of replacing it: y = x + F(x). The local slope becomes 1 + F′(x), so there’s always a route for the gradient with a factor of exactly 1, an express lane from the loss back to the earliest layers. Every transformer layer is built this way (Part 4).
• Normalization. Normalization layers (LayerNorm, or RMSNorm in many recent LLMs) rescale vectors to a standard size inside every layer; in most current LLMs, the vector going into each attention and feed-forward block. However large or small the running values get, each block works on well-scaled numbers, so its local slopes stay in a sensible range instead of compounding layer after layer.
• Gradient clipping. Before each step, measure the length of the whole gradient, all the weights’ gradients taken as one long vector. If it’s over a threshold (1.0 is a common choice), scale it down to that length. The direction is kept and the size is capped. It’s a seatbelt for exploding gradients, not a cure, and it does nothing for vanishing ones.

Overfitting: memorizing instead of learning

Gradient descent lowers the loss on the data it’s given, but you want a model that does well on data it has never seen. Part 5’s Fig. 4 showed the two coming apart: the large model’s held-out loss turned upward while its training loss kept falling, because memorizing was the cheapest way to keep lowering it. Here is the same failure at its smallest: a curve with 12 adjustable weights trains on 12 noisy points. Press play, watch both loss curves, and decide when you would have stopped.

The training loss falls the whole way down; by the end the curve passes almost exactly through all twelve points. The validation loss, measured on 100 held-out points the model never trains on, tells the real story. It bottoms out early, around step 250, then climbs to about three times that, while the curve bends itself through every training point and swings wildly where they’re sparse, in the wide gap on the right. Every step after the green dot made the model better at its twelve points and worse at everything else.

That’s why serious training runs hold out a validation set (Part 5’s held-out line) and watch its loss. The two standard responses are to keep the checkpoint where validation loss was lowest (early stopping), and to add regularization, anything that discourages memorizing. Weight decay is one. Bending a curve exactly through twelve noisy points takes large weights, and weight decay makes large weights expensive. Set it to Medium: validation loss now ends at 0.18 instead of 0.51, and the wild swing is gone. Push it to Heavy, though, and the whole curve gets squashed toward zero: the model can’t even fit the real pattern, and both losses stay high. That’s underfitting: regularization is a dial too. Dropout (randomly switching off parts of the network during training) and simply having more data are the other usual remedies.

How much does this matter for LLMs? Less in pre-training than you might expect. A model trained on trillions of tokens sees most of them only once or a few times, which leaves little room to memorize, except for text that appears over and over (the memorization risk from Part 5). It matters a lot in fine-tuning, where a few thousand examples go round several times.

Pre-training plus backprop produces a base modelBase modelModel after pre-training only. Good at continuing text, not yet an assistant.: a superb autocomplete that isn’t yet an assistant. The next part keeps exactly this machinery, the same loss, backprop and AdamW, and changes only the data.