Large Language Models· Lesson 3 of 7
- 1How an LLM Works Inside8 min
- 2The Context Window — the Model's Short-Term Memory7 min
- 3Temperature & Sampling — Controlling Randomness7 min
- 4System vs User Messages — Who's Speaking7 min
- 5Choosing a Model — Bigger Isn't Always Better8 min
- 6What LLMs Can't Do7 min
- 7Cost & Latency — How Your Bill Adds Up7 min
Temperature & Sampling — Controlling Randomness
In one line · The model doesn't always pick the top token; temperature controls how much it gambles on a less-likely one.
High temperature = creativity and variety (and more hallucination); low = consistency. Choosing it separates a writing assistant from a data extractor.
Like a roulette wheel: low temperature lands on nearly the same number each time; high spins wider and surprises you.
The name comes from real heat: hot molecules jitter around randomly, cold ones settle in place. The model behaves the same — hotter means more randomness, colder means more consistency.
A temperature near 0 returns nearly the same answer every time (good for classification and extraction); a higher one (around 0.8–1) gives varied phrasings each time (good for ideation and writing).
At every step the model has a probability distribution over all possible tokens. Temperature is applied to that distribution before picking: below 1 it sharpens the differences — high probabilities get higher and low ones get crushed, until temperature = 0 makes the pick effectively deterministic (always the top token). Above 1 it flattens the distribution, giving rare tokens a real chance. Other sampling knobs work alongside it, like top_p, which cuts off the distribution's tail before picking — the usual advice: tune one of them, not both at once.
The exact same call — the only difference is the temperature value, and the product behavior changes completely:
# Data extractor: we want the same output every time
extract = client.chat.completions.create(
model="gpt-4o-mini",
temperature=0,
messages=[{"role": "user", "content": "Extract the order ID from: ..."}],
)
# Idea generator: we want real variety
ideas = client.chat.completions.create(
model="gpt-4o-mini",
temperature=0.9,
messages=[{"role": "user", "content": "Give me 5 taglines for a coffee brand"}],
)🧪 Try it yourself — run one prompt across temperatures: take a single prompt (say: 'suggest a name for a coffee-delivery app') and run it 5 times at temperature 0, 5 at 0.7, and 5 at 1.2. Log the outputs in a table and compare: how often did each group repeat itself? Where did output start getting weird or incoherent? Then flip the experiment with an extraction prompt ('pull the order ID from this text') and see why high temperature is a disaster there.
- Leaving the default for everything. JSON extraction at high temperature gives you different keys each run; brainstorming at 0 returns the same 3 ideas.
- Believing temperature = 0 means byte-identical output every time. It gets very close, but things like parallel GPU arithmetic can cause tiny variations — don't build tests that diff text literally.
- Treating temperature as the hallucination fix. Low values reduce variance, but a model confidently wrong about a fact stays confidently wrong at temperature 0.
Turn it near 0 when correctness has one shape: classification, data extraction, JSON output, support answers that must be consistent, anything measured in an eval. Turn it up (0.7–1) when variety is the point: ideation, marketing copy, dialogue, alternative names.
Don't reach for it as the first response to any quality problem — if answers are wrong on substance, the issue is usually the prompt, the context, or the model choice, and temperature won't fix it. Also note: some recent models (especially reasoning models) ignore the parameter or pin it — check the model's docs before assuming it applies.