Quantization: Squeezing a Model Into Fewer Bits

Llama 3 8B is eight billion numbers. Stored the standard way, two bytes apiece, that’s 16 GB, which is more than a 16 GB laptop has to spare once the operating system has taken its share. Yet models of exactly that size run on exactly that kind of laptop every day, at a perfectly usable speed.

The trick sounds like vandalism. Take nearly every one of those eight billion carefully trained numbers and round it to a 4-bit code. Not four digits: four bits, so at most sixteen possible values, about as coarse as a ruler with sixteen marks. (Each small group of weights gets its own ruler, and that detail turns out to matter a lot.) The file shrinks to under a third of its size, and the model still answers nearly as well as before.

Why doesn’t that wreck it? Where does it finally break? And before you download 40 GB, how do you work out whether a model will fit on your machine? The answers start with a little counting.

Fewer bits, fewer possible values

n bits can make 2ⁿ different patterns, so they can hold at most 2ⁿ distinct values: 65,536 for 16 bits, 256 for 8, just 16 for 4. Every bit you remove halves the menu.

The weightsWeightsThe learned numbers that scale inputs inside the network. Set during training, frozen at inference. of a trained model (Part 3) are usually stored as 16-bit floating-point numbers (the formats are called fp16 and bf16), two bytes each. That makes the memory bill easy to estimate:

QuantizationQuantizationStoring weights with fewer bits (8, 4…) to save memory and speed up inference, at some cost in accuracy. stores the weights with fewer bits, typically 8 or 4, by rounding each one to the nearest value on a coarser grid. Halve the bits and you halve the memory. The price is that every weight is now slightly wrong. Fewer bits mean bigger errors, and if you push too far the model degrades badly. Most of this post is about why “slightly wrong” is so often good enough, and how to tell when it isn’t.

Rounding a weight by hand

Here’s the simplest scheme that works well: symmetric 4-bit integers, “int4” for short. A 4-bit integer holds the whole numbers from −8 to 7. Weights aren’t whole numbers; they’re fractions like 0.23 or −0.41. (Real LLM weights are often smaller still, a few hundredths, but the method works the same at any size.) So we stretch the integer ruler to fit the weights.

Take a small block of weights and find the largest one by size, ignoring its sign. Say it’s 0.8. Map it to the end of the ruler, 7. That fixes the distance between neighboring marks, which is called the scale:

Each weight is then stored as an integer: divide it by s and round. To use the weight later, multiply back. Two weights from this block, worked in full:

• 0.23 ÷ 0.1143 = 2.01, which rounds to 2, stored as the bit pattern 0010. It comes back as 2 × 0.1143 = 0.2286, off by about 0.0014.
• −0.8 ÷ s = −7 exactly (s is 0.8 ÷ 7, so this one can’t miss), stored as 1001. It comes back as −7 × s = −0.8, with no error at all.

Two facts fall out of this. The block’s largest weight always comes back exactly, because it set the scale. And no weight can be off by more than half a step, s/2 ≈ 0.057, because rounding never moves a number more than halfway to the next mark. (You may have spotted that −8 never gets used: the ruler runs from −7 to 7, so the largest weight lands on an end mark whatever its sign. Some formats, such as llama.cpp’s Q4_0, use the sixteenth code by mapping the largest weight to −8.)

Twenty-seven. At 2 bits the step is 0.8, so anything smaller than 0.4 rounds to zero, and in a block shaped like this, where most weights are small, that’s nearly all of them. The block is mostly erased.

Now walk back up. Each bit you add roughly halves the step, and the errors shrink with it. At 4 bits the average error is 0.032, about a tenth of a typical weight in this block. At 8 bits it’s 0.0014 and the red slivers vanish: 255 levels make a very fine ruler, which is why 8-bit quantization is usually close to lossless. Also look at the small print under “Step s”: 32 weights at 4 bits take 144 bits, because the block’s 16-bit scale has to be stored too. That’s 4.5 bits per weight, not 4. Hold on to that number.

One big weight spoils its block

Why bother with small blocks? You could compute one scale for a whole weight matrix, millions of weights, and the scale would cost next to nothing per weight.

The trouble is what sets the scale: the single largest weight. Trained weights mostly cluster near zero, but a few are much bigger than the rest, and one of those stretches the ruler for everyone who shares its scale. The marks spread apart to make room for the giant, and the small weights, which are most of them, fall into the gap around zero.

With the outlier and one shared scale, 155 of the 256 weights round to zero, and the rounding noise jumps from 12% of a typical weight to 50%. One number did that. Shrink the blocks to 32 and the damage is fenced into the outlier’s own block: everywhere else the step drops back to about 0.06, and the noise falls to 22%. With no outlier at all it would be 9%, so most of what’s left comes from that one spoiled block. Smaller blocks do better still, but look at the cost along the bottom of the chart: every block carries its own 16-bit scale, so blocks of 8 spend 2 extra bits on every weight. Spent instead on 5-bit codes in blocks of 16, the same 6 bits per weight would cut the noise to 6%, less than half what blocks of 8 manage.

Practical schemes settle in between. The common choice is the one you just used: blocks of 32 weights, one 16-bit scale each. That adds 16 ÷ 32 = 0.5 bits per weight, so “4-bit” with blocks of 32 really means about 4.5 bits per weight. (Formats with blocks of 128 pay a little over 4.1.)

Will it fit? Doing the memory math

Now the payoff. Here is the formula applied to an 8-billion-parameter model like Llama 3 8B:

Why does 4-bit land near 5 GB rather than 4? Two reasons, and you’ve already met the first. The block scales add half a bit per weight: 4.5 bits, so about 4.5 GB. Second, some layers are more sensitive to rounding than others, so practical recipes often keep them at higher precision. The usual suspects are the embedding table, which turns token IDs into vectors, and the output layer, which turns the final vector into a score for every token. In Llama 3 8B, each of those holds 4,096 numbers for each of its 128,256 tokens: 128,256 × 4,096 ≈ 525 million weights, about a billion for the pair. Keep them at 8 bits instead of 4.5 and they cost an extra half a gigabyte: about 5 GB in total.

The weights aren’t the only thing in memory, though. While a model generates, it keeps the KV cacheKV cacheStored attention keys and values from earlier tokens. Speeds up generation within a request.: the attention keys and values of every token in the context, saved so they aren’t recomputed at each step. In Llama 3 8B that’s the 131,072 bytes per token you worked out in Part 4: 2 (a key and a value) × 32 layers × 8 shared key-value heads × 128 numbers × 2 bytes. That’s nothing for one token and a lot for the 128k tokens (131,072) that Llama 3.1 8B, the same shape with a longer window, can hold. Every conversation served at the same time needs its own cache. And on top of all that come the activations (the intermediate results of the current step) and the runtime’s own overhead.

At 128k tokens the cache weighs 17.2 GB, more than three times the 4-bit weights, and quantizing the weights did nothing for it. (Some engines can store the KV cache in 8 or 4 bits as well: the same trick, applied to a different pile of numbers.) At 8 bits the 70B model fits an 80 GB card with a 16k context, but at 32k this calculator says it misses by about 0.2 GB, close enough that the real answer depends on your engine. At 4 bits there’s room to spare. Now switch back to the 8B model, set the context to 8k and serve 16 conversations at once: 16 × 8,192 = 131,072 tokens, the same 17.2 GB of cache as one 128k conversation.

One last trap before you compare any of this with your hardware: units. The formula counts billions of bytes, but memory chips come in powers of two, so a “16 GB” laptop and a “24 GB” GPU really hold 16 and 24 gibibytes (GiB, 2³⁰ bytes each). The 16-bit Llama 3 8B, 16 × 10⁹ bytes, is 16 GB but only about 14.9 GiB. Some tools count in GiB while still printing “GB”, so the same model looks 7% smaller in one window than in another. Flip the Units switch in Fig. 3: every number shrinks by 7%, and nothing that fit stops fitting. Pick one unit and convert everything to it before you compare.

Smaller is also faster

Here’s a puzzle. A 4-bit model has to unpack every weight and multiply it by its block’s scale before using it, which is extra work. Yet it usually generates text faster than the 16-bit original. Why? To produce one token, the model runs once from bottom to top, and every weight has to travel from memory to the chip’s arithmetic units. For a single conversation that trip, not the arithmetic, is the bottleneck: Part 3 calls the decode phase memory-bandwidth bound. So there’s a simple ceiling on speed:

Take an illustrative 1,000 GB/s of bandwidth, the class of a high-end graphics card. With 16 GB of weights the ceiling is about 62 tokens per second; with 5 GB it’s about 200. Fewer bytes per weight, more tokens per second.

It’s a ceiling, not a promise. The 4-bit codes have to be unpacked and multiplied by their scales on the fly, which costs some compute. And the gain shrinks where arithmetic is the bottleneck instead of memory: reading a long prompt (the prefill), or serving many conversations at once. That’s where hardware with native low-precision arithmetic, such as the FP8 units on recent data-center GPUs, earns its keep, provided the activations are rounded to 8 bits too, not just the weights.

How much does rounding hurt?

Each weight’s error is small, but there are billions of them, and their errors flow through dozens of layers, each feeding the next. Whether they cancel out or pile up into a different answer isn’t something you can read off any single weight. You have to run the model and look.

So let’s run one. The figure below holds a real neural network, tiny by LLM standards at 1,185 parameters, trained to say which of two interleaved spirals a point belongs to. The task needs precise boundaries, so it shows damage quickly. Its weights are rounded live with the same scheme as Fig. 1 and Fig. 2, blocks of 32 and all (its 65 biases stay at full precision, as is common).

At 8 bits the test scores don’t budge and only hairline slivers of the map flip. At 6, only C slips: it loses a single test point of 400. At 5 bits A is still perfect, while B misses 9 points and C misses 6. At 4 the three part ways: A makes its first mistakes but still gets 99% of the test points right, B gets 92% and C 81%, though all three came out of the same recipe, trained from different random starting weights. At 3 bits all of them fall apart, and at 2 they do about as well as a coin toss.

Why does it hold up so well, then collapse so fast? The network answers by computing a score and checking its sign, so a point flips only if rounding pushes its score across zero: what matters is the nudge compared with the margin. At 6 bits, rounding moves network C’s scores by about 7% of their typical size, while half the test points score beyond ±9.4, so nothing happens to them. The one point C loses was a near-tie to begin with: its score went from −0.75 to +1.4. At 4 bits the nudges grow to between 35% (A) and 83% (C) of a typical score, and even confident points flip. An LLM faces the same test at every token: when the next word wins by a mile, a little noise changes nothing, but when two candidates are nearly tied, it can tip the choice.

LLMs show the same shape, and the practical rule of thumb is: 8-bit is usually near-lossless, 4-bit is the common sweet spot, and at 3 bits and below quality often drops sharply, especially for small models and for math or code. Why small models? A common explanation is that they have less spare capacity to absorb the damage. Math and code are less forgiving: a flipped near-tie in prose usually still reads fine, but in code one wrong digit or bracket breaks the whole answer.

The toolbox: PTQ, QAT and tiny floats

Everything so far has been post-training quantization (PTQ): take a finished model and round its weights, with no retraining. The simplest form is the round-to-nearest you’ve been doing. Two well-known methods do better by watching the model at work on a small calibration set of sample inputs:

• GPTQ rounds a layer’s weights one column at a time, and after each step adjusts the weights it hasn’t rounded yet to cancel the error it just introduced. Say two weights feed the same output, and their inputs always arrive nearly equal. If rounding pushes the first one down by 0.05, nudging the second up by 0.05 before rounding it cancels the damage to the output. GPTQ does that bookkeeping for a whole layer, using the calibration inputs to learn which inputs move together.
• AWQ (activation-aware weight quantization) notices that a small share of weights matter far more than the rest: the ones that multiply consistently large inputs, which amplify their errors. It scales those weights up before rounding, giving them finer steps relative to their size, and scales their inputs down to match, so the layer computes the same thing with its important weights protected.

The files you’ll most often download to run a model locally are neither. GGUFGGUF / GPTQ / AWQCommon formats and methods for quantized models. is the file format of llama.cpp, the engine inside many local-model apps. Its standard recipes are PTQ computed from the weights alone, no calibration needed (an optional “importance matrix” built from sample text can refine them), and its type names spell them out. Q8_0 is plain round-to-nearest: 8 bits in blocks of 32 with one 16-bit scale each, 8.5 bits per weight, Fig. 1’s scheme at 8 bits. Q4_K_M is a 4-bit “k-quant”: each block of 32 stores a scale and a minimum (an offset, so its grid needn’t be centered on zero), the blocks are grouped into super-blocks of 256 whose scales and minimums are themselves quantized to 6 bits, and the M (medium) mix keeps some sensitive tensors at higher precision. It averages a little under 5 bits per weight, which is how an 8B model becomes a file of about 5 GB.

The second family is quantization-aware training (QAT). Instead of rounding a finished model and hoping, you train it, or briefly fine-tune it, with the rounding simulated in every forward pass, so the weights settle into values that survive it. There’s a wrinkle. Rounding is a staircase, flat almost everywhere, so its gradient is zero and backpropagation (Part 6) would learn nothing through it. The standard fix is to pretend, on the backward pass only, that the rounding isn’t there. QAT usually gives the best quality at low bit-widths, but it costs a training run, so it’s mostly done by the people who make the model.

The third idea changes the grid itself. Integers are evenly spaced, but weights aren’t evenly spread: most are small. Low-precision floating-point formats put more of their levels near zero. A float is scientific notation in binary: a few significant bits (the mantissa) times a power of two (the exponent). Small exponents give closely spaced values near zero and large ones give widely spaced values far out, so in effect every number carries its own little scale. FP8 fits a sign, an exponent and a mantissa into 8 bits (two common layouts, E4M3 and E5M2, trade precision for range), and recent data-center GPUs compute in it natively. FP4 (the layout called E2M1) has just sixteen codes, ±0, 0.5, 1, 1.5, 2, 3, 4 and 6, and is used with a scale per small block, just like int4. Look at the spacing: half-steps near zero, a step of 2 at the far end.

On the flat block FP4 loses badly, with 69% more error: its wide outer steps sit right where most of the values are. On Fig. 1’s block, many small weights plus 0.72 and −0.8, FP4 has 15% less error, and with an outlier its lead grows to 34%. Neither grid is better everywhere; each is a bet on where the values are. Fig. 1’s block is a lucky one for FP4, though. Over thousands of random bell-shaped blocks of 32, int4 comes out slightly ahead (FP4 has about 4% more error and wins about 4 blocks in 10), while on heavy-tailed blocks, a few large values among many tiny ones, FP4 wins about 7 in 10. Trained networks often have tails like that. The 4-bit base model in QLoRAQLoRALoRA fine-tuning on top of a 4-bit quantized base model. (Part 9) makes the same kind of bet with its own format, NF4, whose sixteen levels are spaced to match a bell curve.

That completes the model itself: pre-trained, shaped, adapted cheaply and now cheap to run. But a trained model is frozen, and it only knows its training data. To answer questions about your private documents or yesterday’s news, or to take actions, it has to be connected to outside systems. The next five parts cover how, starting with turning text into vectors you can search by meaning.