What an LLM Actually Does (and Why It Remembers Nothing)

Above your phone’s keyboard sits a strip of three guesses. Type “see you” and it offers tomorrow, soon, later. Now try a small experiment: keep tapping the middle suggestion. You get a sentence, sort of. It’s grammatical for a few words and then it wanders off. The keyboard is asking a perfectly good question, what usually comes next?, but it’s a small model that sees only a short stretch of what you typed, so it loses the thread.

Here is the claim of this post. A large language model plays exactly that game. Same question, but the guesser was trained on trillions of words and looks back over everything written so far, up to hundreds of pages of it. That’s what turns “a sentence, sort of” into working code, a contract summary or a decent haiku. There’s no lookup table of answers inside, and (this is the part that surprises people) no memory of you.

That sounds wrong. You told a chatbot your name an hour ago and it still uses it. So where was your name kept? The answer falls out of the same game, and so do two everyday oddities: the same question getting two different answers, and long chats turning forgetful and expensive.

The whole trick: guess the next piece, then do it again

A large language model (LLMLLMLarge language model. Generative model trained to predict the next token.) is a generative model trained on a huge amount of text. Strip away the chat window and it does one thing: you give it some text, and it predicts what comes next. The program around it appends that prediction to the text and asks again. And again, until the answer is finished.

Two refinements before we watch it work. First, “what comes next” isn’t quite a word. Models work in tokensTokenA chunk of text (about ¾ of an English word on average) with an integer ID. The unit of processing and billing.: chunks of text that are often a whole common word and sometimes a piece of a rarer one (Part 2 is all about them). Second, the model doesn’t output a token. It outputs a probability for every token in its vocabulary, tens of thousands of them and often over a hundred thousand. Together those probabilities, one per token and adding up to 1, are a probability distribution over the vocabulary. Picking one token from it is a separate step.

The answers are: once per token, and all of it, every time. Each run reads the entire text so far, including the tokens it produced a moment ago, and adds exactly one more. Feeding a model its own output like this is called autoregressive generation. (“Auto” as in self: each prediction is built on the model’s own earlier output.)

Look at the third run of the greedy walk (4/7 on the step counter). Once “Ray” is written, “leigh” comes in at 96%. The model is continuing text one chunk at a time, and some chunks are halves of words.

Then switch to the dice. Same model, same prompt, same probabilities at the first step, but the dice land on “air”, the second favorite, and from there the sentence goes somewhere greedy never went. Every pick is final. The model can’t go back and edit; each later step has to continue whatever is already on the page. That’s why one early unlucky token can steer a whole answer.

Notice too the “chance of this exact text” readout: even the greedy sentence would come out exactly that way only about 1 time in 13 if you sampled it, and the dice sentence about 1 in 3,800. Most long answers are, in this sense, one of a kind.

How does it stop? The vocabulary includes a special end-of-sequence token. When the sampler picks it, generation stops. Another common way to stop is a length limit, such as the max_tokens you set in an API call (APIs also let you name stop sequences). If you’ve ever seen an answer cut off mid-sentence, it most likely hit the length limit.

Where does such a good guesser come from? Training. The model was shown trillions of tokens of text. At every position it guessed the next token, was shown the real one, and had its numbers nudged so the real one would come out a little more likely (Part 5 and Part 6 take that apart). Guessing well across that much text forces it to pick up grammar, facts and the habits of code. It also explains the missing lookup table. Llama 3 8B is 8 billion numbers, about 16 GB at two bytes each, and it was trained on over 15 trillion tokens, roughly 60 TB of text. It can recite passages it saw many times, but it can’t have stored the text, only whatever helps predict it.

Who rolls the dice

Open the box one level. At each step, the model’s last layer produces one raw score per token in its vocabulary. These scores are called logitsLogitRaw, unnormalized score for each vocabulary token before softmax.. They can be any number, positive or negative, and bigger means “more likely”. To turn them into probabilities you need numbers that are positive and add up to 1. The standard trick is softmaxSoftmaxFunction that turns scores into probabilities that add up to 1.: raise e to the power of each score, then divide each result by the total.

Do it by hand with three tokens. Say the logits are 2 for “Python”, 1 for “Rust” and 0 for “COBOL”. Then e² ≈ 7.39, e¹ ≈ 2.72 and e⁰ = 1, which add up to 11.11. Divide: 7.39 / 11.11 = 0.665, 2.72 / 11.11 = 0.245, 1 / 11.11 = 0.090. The exponential does two jobs at once. It makes everything positive, and it turns differences in score into ratios of probability: every extra point of logit multiplies a token’s probability, relative to any other token’s, by e ≈ 2.7. “Python” beats “Rust” by one point, so it’s 2.7 times as likely.

Now the model has handed over its scores, and some code that is not the model has to turn them into a choice. That code is the sampler, and its strategies are the knobs you see in every LLM API:

• Greedy: always take the most likely token. Predictable, and prone to bland, repetitive text.
• Temperature: divide every logit by a number T before the softmax.
• Top-k: sample only from the k most likely tokens.
• Top-p, also called nucleus sampling: sample only from the smallest set of top tokens whose probabilities add up to at least p.

Here are the same three logits at three temperatures. At T = 0.5 the logits double to 4, 2, 0, and the favorite runs away with it. At T = 2 they halve to 1, 0.5, 0, and the long shot becomes more than ten times likelier than it was at 0.5.

T = 1 leaves the model’s own distribution untouched. Notice that temperature never changes the order of the tokens, only how far apart their chances are.

Top-k and top-p cut off the tail. Why bother, if tail tokens are unlikely anyway? Because there are so many of them. Say the 20 likeliest tokens hold 95% of the probability and the other 100,000 share the last 5%. Each of those is a one-in-two-million shot, but together they come up about once every 20 tokens, several times a paragraph. Most are merely odd; some are garbage, and every pick is final. Cutting the tail and then renormalizing (dividing the survivors by their total, so they add up to 1 again) removes that chance.

On the open-ended prompt, top-p at 0.9 keeps six tokens. On the factual prompt it keeps exactly one: “Paris”. That’s the point of top-p: it adapts. When the model is confident, the nucleus shrinks to one or two tokens; when many continuations are reasonable, it widens. Top-k keeps the same number no matter what, which is too many when the model is sure (with k = 5, “not” and “located” stay in the running after “The capital of France is”) and can be too few when it isn’t.

Two more things are worth finding in that figure. Drag T toward 0.1 and the bars collapse onto the favorite: that’s greedy decoding in all but name, which is why APIs treat “temperature 0” as greedy. Then put T at 2.5 on the factual prompt, with top-k and top-p off, and draw a hundred times. Paris, the right answer, falls to 47%, and Berlin, at about 2%, will probably show up at least once. A high temperature doesn’t make the model wiser or more creative so much as more willing to say unlikely things, wrong ones included. And the toy is kind to it: it has only ten candidates. Over a real vocabulary of 100,000 tokens the flattened tail is enormous, which is why a very high temperature without a top-k or top-p cut can dissolve into gibberish.

The model remembers nothing

Back to the puzzle from the top. You tell a chatbot your name in the first message, and ten messages later it still uses it. Where was your name stored?

Not in the model. A trained model is a very large file of numbers, its weightsWeightsThe learned numbers that scale inputs inside the network. Set during training, frozen at inference. (Part 3), and they don’t change when you talk to it. The same weights serve a huge number of people’s conversations at the same time. If chatting with you changed them, your chat would leak into everyone else’s. (A provider may, if its policy allows, use chats to train a future model. That’s a separate training run, done later, and it isn’t how a chat remembers you.)

So the model is statelessStatelessKeeps no memory between calls. The full context must be sent every time.: it keeps nothing between requests. Text goes in, text comes out, and afterwards it’s as if the request never happened. The way a chat app builds a conversation on top of that is almost embarrassingly simple: every time you send a message, the app sends the entire conversation so far along with it. The model reads the whole transcript as fresh text, every single time, and continues it. A chat, from the model’s side, is just a document that ends where the assistant is supposed to speak next.

Picture a consultant who sees thousands of clients a day and keeps no notes. Every time you visit, you hand over the complete case file, including notes on all your earlier visits, and they read it front to back before answering. They know a huge amount in general (that’s what the weights hold), but everything they know about you is in the file you just handed them. Where the picture breaks: even a consultant with no notes would recognize your face on the way out. The model doesn’t get even that. Once it has answered, nothing of the visit remains, not even the fact that you came.

When a product seems to remember you from last week, it’s the same trick with one extra step: the app saved some notes about you as ordinary data, and slips them into the file. My study notes, the ones this whole series is built on, have a worked example of exactly this. Step through it once, then delete the saved note and replay. Before you do, predict: what will change inside the model?

Nothing inside the model changed. Look at the request the app built in step 5: one block of text with your saved preference sitting inside it. The model didn’t remember that you like Python. It read that you like Python, a moment before answering. Delete the note and the same model, with the same weights, has no reason to prefer Python; in the figure it reaches for Bash. Only the input changed, and the answer followed it.

Real apps format this differently (memories often go inside the system prompt, and special tokens mark who said what), but it all arrives as one sequence of tokens. “Memory” features work this way at scale: notes live outside the model, and the relevant ones are found and inserted into the prompt, often with the retrieval methods from Part 11 and Part 12.

The flip side is privacy, and the mechanism makes it sharper than it looks. A note like “user is job hunting” is a row in someone’s database. And because memory works by pasting notes into the prompt, that row also travels to whoever runs the model, with every request it’s relevant to, and falls under their retention policy too. So it needs ordinary data care: users should be able to see it, correct it and delete it, know how long it’s kept, and have agreed to it being kept at all.

What about the KV cache?

If you’ve read about how LLMs are served, you may have met the KV cacheKV cacheStored attention keys and values from earlier tokens. Speeds up generation within a request. and wondered whether that is memory. It isn’t, and the difference is instructive.

Within a single request, every new token needs certain intermediate results for all the tokens before it (attention “keys” and “values”, which Part 4 explains). Recomputing them for the whole text at every step would be wasteful, so the server keeps them while the request runs. That’s the KV cache, and it’s why the “read everything, every time” loop in Fig. 1 is cheaper than it sounds: every earlier token is still looked at on each step, but its keys and values are computed only once.

Here’s a test for whether something is memory: could it make tomorrow’s answer different? The cache can’t. Everything in it is computed from the text of the request, so the same text always rebuilds the same cache (up to the floating-point wobble from earlier). It makes answers faster, never different. Usually it’s thrown away when the request ends. Some providers keep it for a few minutes so that a later request starting with exactly the same text can skip that work (that’s prompt caching, below), but even then it’s only used when you send that text again.

Everything has to fit on one desk

If the whole conversation is re-sent every time, something has to give eventually. It does. Everything the model can see in one call must fit in its context windowContext windowMaximum number of tokens a model can process in one call (input + output).: a fixed budget of tokens, such as 128k or 200k depending on the model (a few now reach a million). The limit comes from how the model was built and trained, and from cost: the KV cache grows with every token, and the work attention does grows faster than the length (Part 4). Think of it as the size of the consultant’s desk. The desk has to hold the system prompt (the app’s standing instructions), the conversation history, any retrieved documents, any tool results, and the model’s own output, because the reply is written onto the same desk and needs room too.

When the conversation outgrows the desk, the app has to do something. Many apps quietly drop the oldest turns; others summarize them, or stop with an error. Watch what dropping does to an instruction you gave early on.

Sixty, not 64: the system prompt (3k), the saved notes (1k) and the room reserved for the reply (4k) come off the top first. Attach the 30k of documents and it drops to 45. The system prompt survives because the app pins it to the front. Your first message gets no such treatment. Its instruction was just another turn, and once it slides off the left edge, the model isn’t ignoring it. This call never contains it, and the model kept nothing from the calls that did.

Why the bill grows like a triangle

Re-sending everything has a second cost, and this one is literal. Providers bill by the token, and every re-sent token counts as input again.

My notes put numbers on it. Say every turn adds about 500 tokens to the conversation, and count each request as carrying every turn so far, the new one included. The first request sends 500. The second sends those 500 again plus 500 new ones: 1,000. The third sends 1,500. By the tenth turn a single request carries 5,000 tokens. Now add up all ten requests:

The sum 1 + 2 + … + n equals n(n + 1)/2. Draw it as bars and you get a staircase with the shape of a triangle, and the n² hiding in that formula is the trouble: double the length of a chat and its total input roughly quadruples.

(Strictly, a turn’s reply is output: it’s billed separately, usually at a higher rate, and only joins the input from the next request on. So the exact input total is somewhat lower. The triangle keeps its shape.)

The figure below has the two standard fixes for the triangle: prompt caching and trimming. Before you flip either one, guess which of them makes the bars shorter.

Only trimming does. Caching leaves every bar exactly as tall, because the two fixes work on opposite sides of the API.

Prompt caching happens at the provider. Look at the staircase: every request begins with exactly the previous request. So the provider can keep the work it already did on that prefix (its KV cache, kept after the request ends) and reuse it when the next request starts the same way. A few minutes is a common retention, and providers usually require a minimum prefix length, often around a thousand tokens. You still send every token, and they still fill the context window, but the cached part is processed faster and usually billed at a discount. The catch is in the word prefix: a hit needs the text to match exactly from the very first token. That’s why, in the figure, trimming with caching on is so disappointing. Dropping the oldest turn changes the start of every request, so the cache stops matching.

Summarizing or trimming happens in your application. Cut the history to the last few turns, or replace older turns with a short summary, and the bars stop growing: the triangle flattens into a slab, and the total grows in a straight line. The price is the one you saw in Fig. 4: whatever you cut is gone for the model. Summaries lose detail, and writing the summary is a model call of its own.

This series follows one path through the subject: what an LLM is and how it works inside (Parts 1–4), how it’s trained and adapted (Parts 5–9), how it’s made cheap enough to run (Part 10), and how it’s connected to outside knowledge and actions (Parts 11–15). Part 16 then puts the whole machine on one page.

Every number in this post was counted in tokens, and I’ve been vague about what a token actually is. That’s next, along with why a model that writes sonnets can stumble over the r’s in “strawberry”.