Weights, Parameters & Inference: A Model Is a Pile of Numbers

Ask a real-estate agent what a house is worth and she’ll often have a number before she has seen the kitchen. A 1,000-square-foot place around here? About $300,000. Ask how she knows and you’ll get a rule of thumb: houses in this area go for about $300 a square foot.

That rule of thumb is a model. It takes an input (the floor area), multiplies it by a number learned from experience (300), and produces a prediction. Scale that idea up past anything you could hold in your head and you have the inside of an LLM. Open the weights file for Llama 3 8B and you won’t find code that writes sonnets, a database of facts or a list of grammar rules. You’ll find about 8 billion numbers like that 300, 16 GB of them, arranged so that the results of some multiplications become the inputs of others.

How can multiplying and adding produce anything more interesting than a straight line? Where do 8 billion numbers actually sit? And why does a chatbot take in your long prompt in a blink, then type its answer one word at a time? All three answers fall out of building the thing from one number up.

A model with one number in it

Write the agent’s rule as a formula. Call the floor area x and the dollars per square foot w:

The number w is called a weightWeightsThe learned numbers that scale inputs inside the network. Set during training, frozen at inference., because it sets how much the input weighs in the answer. The weight is the model. The formula never changes. What makes this the agent’s model, and not one for a city where houses go for $900 a square foot, is the value 300.

Where does the 300 come from? From data: she has watched a lot of houses sell. Below are twelve sales (made up for this post, with realistic numbers). The model draws a straight line through zero with slope w, and each house’s miss is the vertical gap between its dot and the line. To score the model as a whole, the figure uses the typical miss: square every miss, average the squares, take the square root. (This is the root-mean-square error. Squaring stops misses in opposite directions from canceling out, and makes big misses count extra.)

No line hits them all. Swing it through every angle and the typical miss never drops below about $66,000. The best w, at the bottom of the valley in the lower chart, is 298: the agent’s rule of thumb was very nearly the best single number these sales allow. (The button finds it with a formula, Σ xy / Σ x². Read it as an average of each house’s price per square foot, weighted by its size squared. That’s why the 650 sq ft house at $508 a square foot barely moves the answer: the 2,900 sq ft house counts twenty times as much.)

That valley is worth remembering, because finding the bottom of an error valley is what training a model means. With one weight you can draw the whole valley and read off the lowest point. With 8 billion weights the valley is a landscape over 8 billion directions that nobody can draw, and the miss is how badly the model predicted the next token rather than a dollar amount. Training has to feel its way downhill one small step at a time. That’s the story of Part 5 and Part 6.

Why can’t one number do better? Look at where the misses point. The best line runs under every house up to 1,700 sq ft and over four of the five bigger ones. Some of that is location: the 650 sq ft place that sold for $330,000 is in one of the best spots on the list. Some of it is that part of every price doesn’t grow with floor area at all. Even a tiny house comes with land, a kitchen and a roof. The model can’t express either. Floor area is all it sees, and its line has to start at zero.

One neuron, more inputs

So let it see more. Add the number of bedrooms and a location score from 1 to 10, each with its own weight, plus one extra number, b:

The b is a bias: a fixed amount added whatever the inputs are. In Fig. 1 the line had to pass through zero, as if a 0 sq ft house would sell for $0; the bias lets it start somewhere else. (Nothing to do with bias in the everyday sense.)

A little machine like this, several inputs, each multiplied by its own weight, summed along with a bias, is the heart of a neuronNeuronUnit computing a weighted sum of inputs plus bias, followed by an activation.. (A full neuron adds one more step, a bend, which is coming up.) The name comes from a loose analogy with brain cells, and the analogy is mostly just a name. There’s no biology in it, only arithmetic.

A prediction before you play. The figure starts as Fig. 1’s model: size weight 300, the other three numbers zero. When you let the fit choose all four numbers together, will the size weight stay near 300, go up, or go down?

Two things happened. The typical miss collapsed from about $66,000 to about $12,000, so the new numbers carried real information. And the size weight dropped, from 300 to about 207. Square footage didn’t get cheaper. The other numbers took over jobs the size weight had been doing. In Fig. 1 the line had to start at zero, so every dollar of every price had to be billed per square foot: the value of the bedrooms (bigger houses here almost always have more of them), and also the part of a price that doesn’t grow with size at all, like a good location or the land under even a small house. A line through zero can only fake that fixed part by tilting steeper. Once bedrooms, location and the bias have numbers of their own, they take their share, and size only has to explain what’s left: about $207 per extra square foot.

That’s true of weights in general: a single weight doesn’t mean much on its own. Its value only makes sense next to all the others it was fitted with.

Notice also what stays put while you click from house to house: the four numbers. The inputs change with every house; the weights are the same for all of them. Inputs that vary, learned numbers that don’t: that’s what using a trained model means.

Why the neurons in between need a bend

Now scale up the way every neural network does, in three steps:

1. Give a neuron several inputs, each with its own weight. (Done.)
2. Put many neurons side by side, all reading the same inputs, each with its own weights. That’s a layerLayerA stage of the network whose output feeds the next stage.. Because their weights differ, each neuron in it can respond to a different pattern.
3. Stack layers, so each layer’s outputs become the next layer’s inputs. Later layers can then combine the patterns that earlier layers picked up.

If every neuron only computes a weighted sum, though, step 3 is a trap. Try the smallest stack there is: one input, two neurons in a row. The first computes h = 2x + 1. The second takes h and computes y = 3h − 4. Substitute: y = 3(2x + 1) − 4 = 6x − 1. Two neurons and four learned numbers, and the result is exactly one neuron with weight 6 and bias −1. Nothing the pair can do is beyond a single neuron.

The same algebra works for whole layers. Write a layer’s weights as a grid W, one row per neuron, and its biases as a list b, so the first layer computes h = W₁x + b₁ and the second y = W₂h + b₂. Substitute one into the other:

That’s one layer, with weights W₂W₁ and bias W₂b₁ + b₂. A hundred layers collapse the same way, one pair at a time. A weighted sum of weighted sums is just another weighted sum, so however deep you stack them, the network can only ever draw a straight line (or, with more inputs, a flat plane).

The fix is small: bend each neuron’s output. Before passing its sum on, the neuron runs it through a non-linear activation functionActivation functionNon-linear function (ReLU, GELU…) applied to a neuron's weighted sum. Lets networks model non-linear patterns., f:

The simplest is ReLU (rectified linear unit): f(z) = max(0, z). Negative sums become 0; positive ones pass through unchanged. It’s a hinge: flat, then a ramp. The figure also offers GELU, a hinge with the corner rounded off. (Fig. 2’s neuron had no activation, and that’s normal: a final neuron whose job is to output a number like a price stays straight. It’s the hidden neurons in between that need to bend.)

To see what the bend buys, here’s a pattern no straight line can follow: price against the age of a house. New builds sell at a premium, mid-century houses are cheapest, and century-old houses with character climb back up. (Made-up data again, in a familiar shape.)

With no activation the answer is: never. Every hidden neuron outputs a straight line in age, the output neuron adds up straight lines, and the figure even prints the single neuron the whole network collapses into. One neuron or eight, the typical miss stays at $51,500, the best any straight line can do.

Switch on ReLU and the same network bends: at 4 neurons the typical miss falls from $51,500 to about $5,700, with the same 13 parameters. Each hidden neuron is now a hinge: zero on one side of its kink, a ramp on the other. Four hinges let the prediction bend at four points; at eight neurons the typical miss is down to about $2,200. GELU does the same with rounded corners. (The figure pins the kinks and fits only the output neuron, exactly; real training moves every weight, kinks included.)

GELU’s exact shape is z × Φ(z), where Φ(z) is the chance that a standard bell-curve variable comes out below z; it dips slightly below zero just before it turns on. GPT-2 and GPT-3 used it. Llama models use SwiGLU, a gated variant: two separate weighted sums of the same input, one passed through a smooth hinge, multiplied together. The shapes differ; every one of them bends.

This is a famous result in miniature, the universal approximation theorem: with enough hidden neurons, even a single hidden layer with a bend like ReLU or GELU can match any continuous curve over a bounded range as closely as you like. It promises that the right weights exist, not that training will find them or that “enough” is a small number. Real networks get their power from depth as well as width: each layer bends and recombines what the layer below found, so complicated patterns get built out of simpler ones.

Counting to eight billion

Every weight and every bias is one number that training has to find; together they’re the model’s parametersParametersAll learned numbers (weights and biases) in a model.. Counting them is easy once you see the pattern. A fully connected layer from n_in inputs to n_out neurons gives each neuron one weight per input, plus its bias:

Fig. 2’s neuron: 3 × 1 + 1 = 4. Fig. 3’s network at 8 hidden neurons: (1 × 8 + 8) + (8 × 1 + 1) = 25. Now one matrix at LLM width, 4,096 numbers in and 4,096 out: 4,096 × 4,096 + 4,096 = 16,781,312, about 16.8 million parameters in one matrix. (Llama’s matrices actually skip the biases, which makes it 16,777,216. At this size the biases are a rounding error.)

What an LLM calls a “layer” is a bundle of several matrices like that one, seven in Llama 3, which Part 4 opens up. Large LLMs stack tens to over a hundred of these layers, each with thousands of neurons. That’s where the billions come from, and it’s what lets them represent patterns far more complex than a house price. The published layer counts: Llama 3 8B has 32 layers, Llama 3 70B has 80, Llama 3.1 405B has 126, and GPT-3 (175B) had 96.

One more guess before you step through the next figure. Besides its layers, Llama 3 8B has two big tables: the embedding table from Part 2, which turns each token into a vector on the way in, and the output layer, which turns the final vector into a score for every token on the way out. What share of its 8 billion numbers do you think sits in those two tables?

About 13%: 1.05 billion of the 8.03 billion. The tables are big because each has a row of 4,096 numbers for every one of the 128,256 tokens in Llama 3’s vocabulary: 525 million apiece. (Switch the figure to 405B and its two tables are only about 1% of the model.) The other 87% of the 8B lives in its 32 layers, and within each layer about four-fifths sits in the feed-forward matrices. Watch the ruler, too. It jumps from 41 parameters (step 3’s little stacked network) to 16.8 million in one step, because a layer’s weights grow with the product of its input and output sizes. Double the width of a square layer and you quadruple its weights.

Parameters, and the numbers people choose

All the learned numbers together, every weight and every bias, are the model’s parameters. “An 8B model” means 8 billion parameters. They’re set during training and then frozen.

A second set of numbers is chosen by people, before training starts: how many layers, how wide each one is, the learning rateLearning rateStep size used when updating weights from gradients. (how big a step training takes each time it adjusts the weights), the batch size (how many examples it looks at per adjustment). These are hyperparameters. The test: if training set it, it’s a parameter; if a person picked it, it’s a hyperparameter. In Fig. 4, “32 layers” and “4,096 wide” are hyperparameters, and the 8,030,261,248 numbers they make room for are parameters.

Inference: running the frozen pile

Once the numbers are frozen, using the finished model to produce answers is called inferenceInferenceUsing a trained, frozen model to generate outputs.. It’s what happens every time you chat with an assistant. The model runs forward: your tokens go in as numbers, get multiplied by weights layer after layer, and out comes a score for every possible next token (Part 1).

How much arithmetic is that? Each parameter takes part in about one multiplication and one addition per token: two floating-point operations, or FLOPs. So a dense model, one that uses every parameter for every token, needs about 2 × N FLOPs per generated token, where N is the parameter count. (Mixture-of-experts models hold many parameters but use only some of them for each token; for those, N is the number actually used. Two smaller caveats: attention also compares each new token with every earlier one, work that grows with the length of the conversation (Part 4) and stays modest for a few thousand tokens; and the input table is only looked up, not multiplied, so 2 × N slightly overstates.)

Sixteen billion operations per token sounds like the bottleneck. Usually it isn’t. At 16 bits (2 bytes) per weight, 8 billion weights fill 16 GB, and the chip can only multiply a weight after it has traveled from memory to the arithmetic units. Take round numbers for an H100-class data-center GPU: memory delivers about 3 TB per second, and the math units do about 1,000 trillion operations per second. To produce one token, an 8B model has to read 16.06 GB (8.03 billion weights at 2 bytes), which takes 5.4 ms, and do 16 billion operations, which take 0.016 ms. The math finishes in a blink, then waits for the next weights to arrive: it’s busy about 0.3% of the time.

Generation has two phases that treat that trip very differently:

• Prefill. The whole prompt is processed in one pass. Every prompt token is known up front, so each weight, once fetched, is used for all of them in parallel. That’s fast per token, and the pass also produces the first token of the answer.
• Decode. The rest of the answer comes out one token at a time, because each token depends on the one before (Part 1’s loop). Every step needs every weight again, fetched from memory to do the arithmetic for just one new token per conversation. That’s slower, and usually limited by memory bandwidth (how many bytes per second memory can deliver), not by arithmetic. (Why not keep the weights next to the math units? The fast memory on the chip itself holds tens of megabytes; 16 GB has to sit in the GPU’s main memory and stream across on every pass.)

In the idealized figure, decode gets four times faster; the prefill doesn’t move. 500 × 16 GFLOPs is 8 trillion operations, 8 ms of math against 5.4 ms of reading, so prefill was waiting on arithmetic, and quartering the bytes doesn’t touch arithmetic. Shrink the bytes and only the phase that was waiting on bytes speeds up. (Real systems gain less than 4×. Practical 4-bit formats also store a scale for every small block of weights and often keep a few sensitive layers at higher precision, so an 8B model comes to about 5 GB, not 4 (Part 10). Each request’s KV cacheKV cacheStored attention keys and values from earlier tokens. Speeds up generation within a request. (Part 4) doesn’t shrink and is read at every step too. And the 4-bit numbers have to be unpacked before use.)

Per token, that’s 16 microseconds for each prompt token and 5.4 milliseconds for each answer token: an output token takes over 300 times as long as an input token. (Drag the prompt below about 330 tokens and prefill turns memory-bound too: reading the weights once then takes longer than all the prompt’s math.)

That’s why LLM speed is quoted as two numbers: time to first token, which is mostly prefill, and tokens per second, which is decode. It’s one reason output tokens usually cost more than input tokens. And it explains a trick that serving systems rely on.

Frozen, and priced by the parameter

Two consequences of a frozen pile of numbers catch people out.

Fig. 4’s layers came in two kinds: attention matrices and the feed-forward matrices that hold most of the parameters. Next, what they actually do, and how the word “bank” figures out whether it’s next to a river or a vault.