LoRA: Steering Billions of Weights by Training a Few Million

Say you have an open model with 8 billion weights, and you want it to answer support tickets in your company’s voice. Part 7 showed the recipe: fine-tuneFine-tuningFurther training of a pre-trained model on a smaller, curated dataset. it on a few thousand examples of good answers. Done the straightforward way, that adjusts every one of the 8 billion weights, and each weight being trained also needs memory for its gradient and optimizer state: about 128 GB in a common setup, more than an 80 GB data-center GPU holds. What you get back is another 16 GB file, nearly identical to the one you started with.

Now do it again for the legal team, and again for the data team: three 16 GB files, each differing from the original by a sliver. Any programmer can feel the waste. We don’t copy a whole repository to record a change; we store the diff.

LoRA, short for low-rank adaptation, takes that instinct seriously, with a twist. It freezes all the original weights and trains a small set of extra numbers that sit on top of them, often under 1% of the model’s size. So little is trained that fine-tuning often fits on a single GPU at a fraction of the cost, which is why it’s usually the method individuals and small teams use when they fine-tune open models. The twist is why so few numbers are enough: the change a task needs tends to have a shape, and that shape can be written down very compactly.

A fine-tune is a diff

Start with one weight matrix: a grid of numbers that turns an input vector into an output vector (Part 3). A transformerTransformerNeural-network architecture built from attention and feed-forward layers. The "T" in GPT. layer is mostly a handful of these. Call one W, with d rows and k columns. In an 8B model the attention’s query matrix is 4,096 × 4,096 = 16,777,216 numbers, and the model has 224 big matrices like it, from a quarter of that size to three and a half times it.

Any fine-tune turns W into W + ΔW: whatever training does, you can write the result as the old weights plus a difference. Full fine-tuning learns every entry of ΔW directly, so the diff is as big as the matrix: 16.7 million trainable numbers here, billions across the model.

LoRA leaves W frozen (no gradients, no optimizer state, never changed) and builds the diff out of two thin matrices instead:

B is tall and thin (d rows, r columns), A is short and wide (r rows, k columns), and r is small: commonly 8 to 64. That r is called the rank; the next section shows why the word fits. Multiply B by A and you get a full d × k grid, the same shape as W, so it can be added on. The α/r in front is a volume knob for the adapter; we’ll come back to it after watching one train.

Count what gets trained. For the 4,096 × 4,096 matrix with r = 8, B holds 4,096 × 8 numbers and A holds 8 × 4,096:

Doubling r doubles the count, since r simply multiplies (d + k). Doubling the matrix is the more interesting one. The full update grows four times, because it’s d × k, but LoRA’s two strips only grow twice, because they’re d + k. So LoRA’s share halves, from 0.39% to 0.20%. The bigger the matrix, the better the deal. The deal also has a limit you can find: put a high rank on a small matrix and the strips stop being thin. At r = 64 on a 512 × 512 matrix they already hold a quarter as many numbers as the matrix, and at r = 256 (half its side) they’d hold just as many.

Careful with the diff analogy, though. A code diff is small because it touches few lines. LoRA’s diff touches every weight: B · A is a full d × k grid, and every cell can be nonzero. It’s small for a different reason: all 16.7 million cells are generated from 65,536 numbers, the way a pattern is generated from a rule. “LoRA changes 0.39% of the weights” is wrong. “LoRA changes all of them, using 0.39% as many numbers” is right.

Which raises the real question. A full update can set each of the 16.7 million cells to anything at all; cells generated from 65,536 numbers can’t be just anything. What can B · A express, and why would that be enough?

How many numbers does a times table need?

Picture a 10 × 10 multiplication table. It has 100 cells, but you don’t need to memorize 100 numbers to rebuild it. The labels along the top and down the side, 1 to 10 each, are enough, because every cell is its row label times its column label. Twenty numbers, a hundred cells.

Any grid built that way, where each cell is (a number for its row) × (a number for its column), is called rank 1. It has a telltale look: every row is the same pattern, scaled up, scaled down or flipped. And it’s exactly what B · A is when r = 1. B is a single column of d row labels, A is a single row of k column labels, and their product is a d × k times table. Inside the layer it has a concrete meaning: A measures how strongly the input contains one particular pattern (a weighted sum, one number), and B adds one particular pattern to the output, scaled by that number. One direction in, one direction out.

With r = 2 you get the sum of two such tables; with r = 8, eight. Now a fact that sounds too good to be true: every matrix can be written as a sum of times tables, and there’s a best way to do it. Pick tables whose patterns don’t overlap at all (every two column patterns are perpendicular, and so are every two row patterns) and sort them from strongest to weakest. Then the first r tables are the closest any r tables can get to the original, measured by adding up the squared differences cell by cell. No other choice of B and A does better. (That sorted breakdown is the singular value decomposition, or SVD, and the “closest possible” part is a theorem, not a hope.) A 4,096 × 4,096 matrix may need all 4,096 tables to be exact. The useful question is how many you need to get close, and that depends entirely on the matrix.

LoRA’s bet is that some change that does the job is close to a sum of only a few tables. A new task usually needs the model to respond to a few new input patterns in a few new ways, not to rewire everything at once. In the jargon, the update it needs is low-rank: a few directions describe it well. (Note the wording: not “the change full fine-tuning would make” but “a change that works”. Those can differ.)

The figure below measures closeness the same way. A change’s energy is the sum of its squared cells, and “captured” is the share of that energy the best rank-r version reproduces.

The focused change is all but captured by three directions, 99.6% of its energy; what’s left in “what’s missed” is faint static. (Energy is a squared measure, so the 0.4% left over is a residual whose typical cell is about 7% the size of the change’s: faint, not gone.) The broad change needs 9 directions for 90% of the energy and 17 for 99%: more, but still a fraction of 48. Random noise is the opposite case. There are no patterns to find. No direction stands out (the strongest holds under 9%, and the shares taper off gradually instead of falling off a cliff), and capturing 90% takes 24 directions, at which point the two strips hold as many numbers as the matrix itself. No shortcut exists for noise.

The shuffle doesn’t move the bars at all. Reordering rows and columns only relabels the layer’s inputs and outputs, so it can’t change the rank. The smooth blobs vanish, but a woven, tartan-like texture remains, because every row is still a mix of the same three patterns. That’s what low rank looks like without a convenient ordering, and it’s closer to how a real weight change looks, since a layer’s rows and columns have no natural order. (Compare it with the noise, which has no weave at all.)

Watching an adapter train

How does training actually run with an adapter in place? Nothing exotic happens. The layer used to compute W · x for an input x. Now it computes W · x + (α/r) · B · (A · x): the input also takes a short detour through the adapter, where A squeezes it down to r numbers and B spreads those back out to full size. BackpropagationBackpropagationAlgorithm that computes how much each weight contributed to the error, using the chain rule, working from output to input. (Part 6) runs as usual, but only A and B get updated. Gradients still flow through the frozen W, because adapters in earlier layers need them; nothing is stored for W.

Two details matter, and the walkthrough below shows both with real numbers on a toy layer. The first is the starting point. A starts small and random, and B starts at exactly zero. So B · A is zero, and at step 0 the model with its adapter computes exactly what the base model does. Training starts from everything the base already knows, not from a random disturbance of it.

It’s A. Each matrix’s gradientGradientThe direction and size of change in the loss with respect to each weight. passes through the other one: A’s gradient gets multiplied by B, which is zero, so on the first step it’s exactly zero. B’s gradient gets multiplied by A, which isn’t zero, so B moves first. Once B is nonzero, both train. (That’s also why you can’t start both at zero: each one’s gradient would be multiplied by the other’s zeros, at every step, and neither would ever move.)

The second detail is the merge. After training, W + (α/r) · B · A is just another d × k matrix. Compute it once, save it in place of W, and the adapter is gone: the model has exactly its original shape, and running it costs exactly what the base model did. That’s what “zero extra inference cost” means. Subtract the same product and you get the original W back (up to rounding at 16-bit precision), which is what makes an adapter a removable, swappable thing rather than a permanent change.

And α? Look at what B · A is: a sum of r times tables. Raise r and you add more of them, so the adapter’s output tends to grow, and a learning rateLearning rateStep size used when updating weights from gradients. tuned at one rank can be too hot at another. Dividing by r is the original paper’s counterweight. Its authors picked α once (equal to the first r they tried) and held it fixed, so a bigger r automatically got a smaller α/r, and they didn’t have to re-tune the learning rate for every rank. Many people instead tie α to r (α = r or 2r), which makes α/r a constant 1 or 2 and simply folds it into the learning rate. In the walkthrough, α = 2 and r = 1, so the adapter’s output is doubled. Either way, with the usual Adam optimizer, turning α up acts roughly like turning up the adapter’s learning rate. (Later work, rank-stabilized LoRA, argues that dividing by r over-corrects at high ranks, so those adapters learn too slowly, and divides by √r instead.)

Why it fits on one GPU

The real payoff of training so few numbers is memory. Part 6 added up what training with Adam holds in the usual mixed-precision recipe: 16 bytes for every number being trained (the number itself in 16 bits, a 2-byte gradient, Adam’s two 4-byte running averages and a 4-byte master copy for precise updates). A frozen number costs only its own 2 bytes: no gradient, no optimizer state. Full fine-tuning trains all of an 8B model’s weights, so it needs 8 billion × 16 bytes, about 128 GB, before counting anything else. LoRA keeps training state only for the adapters, a fraction of a percent of the model, so almost all of that disappears. What’s left is dominated by the frozen weights themselves: 16 GB.

What LoRA does not save is arithmetic. Every token still goes forward through all 8 billion weights, and backpropagation still sends the error back through all of them to reach the adapters in earlier layers. What disappears is computing a gradient for each frozen weight, roughly a third of a training step’s work. So “a fraction of the cost” mostly means fewer and smaller GPUs, not 99% less computing.

That leaves one big item, and it’s the one QLoRAQLoRALoRA fine-tuning on top of a 4-bit quantized base model. goes after. QLoRA stores the frozen base’s weights in 4 bits instead of 16, using quantizationQuantizationStoring weights with fewer bits (8, 4…) to save memory and speed up inference, at some cost in accuracy., the subject of Part 10. The adapters stay in 16-bit. During training, each layer’s 4-bit weights are expanded back to 16-bit just long enough to compute with, and gradients flow through them to the adapters as before. The bulk of the base shrinks to about a quarter of its size.

Before activations (more on those in a moment), at 8B full fine-tuning (128 GB) doesn’t fit even on an 80 GB card, while LoRA (about 16 GB) and QLoRA (about 6 GB) both fit on a 24 GB one with room to spare. QLoRA’s 6 GB is mostly 3.6 GB of 4-bit layers plus 2.1 GB for the embedding and output tables, which QLoRA usually leaves at 16 bits. (That’s why it’s more than the 5 GB you’ll see for a 4-bit 8B model in Part 10, which keeps those tables at 8 bits.)

At 70B, full fine-tuning needs over a terabyte. Plain LoRA still needs about 142 GB, because the frozen 16-bit weights alone are 141 GB (Llama 3 70B’s exact count is 70.6 billion, not a round 70). QLoRA needs about 40 GB, which leaves about 8 GB of a single 48 GB card for activations: tight, but workable with short sequences and small batches. That’s the headline of the 2023 QLoRA paper (Dettmers and colleagues) in miniature: they fine-tuned a 65-billion-parameter model on one 48 GB GPU.

Notice what the rank slider does for the 8B model on attention layers: almost nothing. Going from r = 4 to r = 64 moves the LoRA total by less than a gigabyte. Memory is dominated by the base, so rank is mostly a question of quality, not of whether the job fits. At the margin it can still tip the balance: at 70B with FFN adapters, r = 64 adds about 12 GB over r = 4 and pushes QLoRA from a 48 GB card to an 80 GB one.

The figure leaves out activations, the intermediate results of the forward pass that backpropagation needs. They grow with batch size and sequence length, can add many gigabytes, and LoRA doesn’t make them much smaller. Activation checkpointing, the Part 6 trick that throws most of them away and recomputes them during the backward pass (many libraries call it gradient checkpointing), trades time for memory. The main point survives: LoRA moves the training state from “every weight” to “the adapter”.

One base, many adapters

When training finishes, the adapter is just the A and B matrices for each adapted layer, saved as a file. Its size follows from the arithmetic in Fig. 1. For an 8B model with r = 8 on the attention layers, that’s about 6.8 million numbers, 13.6 MB at 2 bytes each. Higher ranks and more target layers push it to hundreds of megabytes. The model it loads onto is 16 GB, and that model never changes. (Here “base model” means whichever frozen model the adapter sits on. That’s often an instruction-tuned model, not only a pre-trained base modelBase modelModel after pre-training only. Good at continuing text, not yet an assistant. from Part 5.)

That size gap changes how you serve fine-tunes. One copy of the base stays in GPU memory, and adapters load on top of it per task, per customer, even per request. Before you play with the figure, predict one thing: twenty customers each want their own fine-tune of this 8B model. How much less storage does one shared base plus twenty adapters need than twenty full copies: 2×, 5×, or nearly 20×?

Nearly 20×: about 16 GB against 321 GB. Twenty adapters like the support-tone one weigh about 270 MB together; the base is everything.

The base switch shows the catch. A base with different shapes fails loudly: the matrices don’t fit, and the loader refuses. Anything with the same shapes loads, and that’s more than newer versions of the same model. Llama 3 8B and Mistral 7B, from different companies, have identical attention and feed-forward shapes, so an adapter for one will typically load onto the other without a word. Adapter files usually record which base they were trained on, but loading code checks shapes, not weights. And ΔW was learned as a correction to one specific W (Fig. 3’s last step): on any other W it’s the right correction to the wrong thing. In a real model it’s worse: every earlier layer changed too, so even the inputs each adapter sees are different.

The storage bars also explain a tension between two of LoRA’s conveniences. Merging an adapter into W is free at inference time, but a merged model is one fine-tune: twenty customers would need twenty 16 GB copies. Serving many adapters from one base means keeping them unmerged and paying for the small detour through A and B on every token. Serve one fine-tune at high volume, and you merge. Serve many, and you keep them separate.

Choosing r, α and where the adapters go

Three settings decide most of what a LoRA run can do.

The rank r. Commonly 8 to 64: the number of directions the update can use (the knob in Fig. 2). Raising it costs little memory (Fig. 4) but enlarges the adapter file, and on a focused task the extra directions find nothing to capture. Too low is the dangerous side; see the pitfall below.

The scale α. Either held constant while you vary r, or tied to r (α = r or 2r) and left alone while you tune the learning rate.

The target layers. These are the matrices that get an adapter. The usual targets are the attention projections, the query, key, value and output matrices from Part 4 (the original LoRA paper’s main experiments adapted just the query and value matrices). Sometimes the feed-forward (FFN) layers get adapters too, which roughly triples the trainable count in an 8B model. The QLoRA paper found that adapting every linear layer, FFN included, was what it took to match full fine-tuning in their experiments; adapting attention alone fell short.

QLoRA leaned on a trick this post took on faith: storing billions of weights in 4 bits instead of 16 without wrecking the model. How that rounding works, and what it costs, is next.