Large Language Models· Lesson 1 of 7
- 1How an LLM Works Inside8 min
- 2The Context Window — the Model's Short-Term Memory7 min
- 3Temperature & Sampling — Controlling Randomness7 min
- 4System vs User Messages — Who's Speaking7 min
- 5Choosing a Model — Bigger Isn't Always Better8 min
- 6What LLMs Can't Do7 min
- 7Cost & Latency — How Your Bill Adds Up7 min
How an LLM Works Inside
In one line · The model predicts the next token over and over — everything it writes is a chain of predictions.
Once you see it as prediction, not 'understanding', its failures become predictable: hallucination, false confidence, sensitivity to phrasing.
Like your phone suggesting the next word — but trained on much of the internet. It looks at what came before, scores the most plausible next token, and picks it.
Picture a vast sea of probabilities; the model walks it step by step, each step choosing the likeliest direction.
You write 'The capital of Egypt is' and it completes 'Cairo' because that's the highest-probability token in context — not because it 'knows' geography.
Your text is first split into tokens — small chunks of text that have IDs in the model's vocabulary. The transformer takes the whole sequence and computes, for every possible token in the vocabulary, the probability that it comes next — so the real output isn't a word, it's a full probability distribution. A sampling step then picks one token from that distribution, appends it to the sequence, and the whole loop repeats: predict → pick → append, until the model emits a 'stop' token. There is no upfront plan for the full reply — each token is generated from everything before it, nothing more.
🧪 Try it yourself — trace the next-token prediction: open any playground that shows logprobs (OpenAI Playground or similar). Type an unfinished sentence like 'The capital of France is', generate exactly one token, and inspect the candidate list with its probabilities. Then change the context — prepend 'In a fantasy novel,' — and watch the same question produce a completely different distribution. What you're looking at is the whole model: a distribution that moves with context.
- Treating the reply as a database lookup. The model generates fresh text every time — it doesn't retrieve a stored answer, which is exactly why it can produce a convincing-looking citation or number that's fabricated.
- Assuming the model 'thinks through' the whole reply before writing. It commits to each token as it picks it, which is why one wrong opening sentence drags the rest along with it.
- Ignoring phrasing sensitivity. Changing one word in the question shifts the entire probability distribution — if results are unstable, try rephrasing before blaming the model.
Return to this picture whenever you're diagnosing odd behavior: why does it hallucinate? Why does rephrasing change the answer? Why does its confidence outrun its knowledge? All of it falls out of 'predict one token after another.' It's also the base of every prompting decision: it tells you what to put in context to shift the probabilities toward the reply you want.
Don't build your app logic on internals — you don't need the transformer math to ship a good product. And don't wield 'it's just prediction' as an argument that models are useless: prediction at this scale produces real capability — the point is knowing its limits, not dismissing it.