Large Language Models· Lesson 2 of 7
- 1How an LLM Works Inside8 min
- 2The Context Window — the Model's Short-Term Memory7 min
- 3Temperature & Sampling — Controlling Randomness7 min
- 4System vs User Messages — Who's Speaking7 min
- 5Choosing a Model — Bigger Isn't Always Better8 min
- 6What LLMs Can't Do7 min
- 7Cost & Latency — How Your Bill Adds Up7 min
The Context Window — the Model's Short-Term Memory
In one line · The context window is the maximum text (in tokens) the model can see at once — your input plus its reply together.
Anything outside the window may as well not exist. It's why a chat 'forgets' the start, and it shapes the design of any long-form app.
Picture a small desk: you pile papers until it's full. Add a new one and an old one falls off. The model is exactly like that.
Like reading a book through a small slot cut in a card: you slide the slot along the lines, and only what's framed inside it exists for you right now — whatever slid past the edge is gone.
A chat recalls your name from 3 messages ago, but after 20 long messages the name scrolled out of the window — so it asks again.
The model is stateless — it remembers nothing between calls. All of a chat's 'memory' is your app resending the whole conversation history with every message, inside the window limit (measured in tokens: system message + history + your new message + room for the reply, all from one budget). When the total exceeds the limit, something has to go — most apps trim the oldest messages or summarize them. And a big window is no magic fix: recall from very long text degrades for content in the middle (the well-known 'lost in the middle' effect), and cost grows with every token you send.
🧪 Try it yourself — fill the context until it overflows: start a fresh chat and tell the model 'My name is Osama — remember it.' Then paste a very long text (a big article or two) and ask for a summary; repeat a few times. Finally ask: 'What's my name?' — with a small-window model or an app that trims history, you'll watch the 'forgetting' moment happen. Also notice when it starts dropping details from the middle of the conversation, not just the start.
- Assuming the model 'remembers' old conversations. There is no memory inside the model — if your app didn't send it in context, it doesn't exist.
- Budgeting in characters or words instead of tokens. Arabic in particular takes more tokens than English for the same meaning, so eyeballing goes wrong fast.
- Forgetting the reply spends the same budget. Fill the window with input and the model will truncate its own answer mid-thought.
Factor the window into your design from day one in anything with long chats, large documents, or accumulating history: it dictates your trimming/summarizing strategy, when you need RAG instead of stuffing the whole document, and the cost budget per call.
Don't over-engineer for it: if your interactions are short — Q&A, classification, extraction from a small text — the window never gets close to its limit, and any 'memory management' layer is wasted machinery. And don't lean on a huge window as a substitute for retrieval when your data outgrows any window or changes constantly.