AI Fundamentals· Lesson 4 of 10
- 1What Even Is AI? AI vs. ML vs. Deep Learning6 min
- 2How Models Learn: Training vs. Inference6 min
- 3What Is an LLM?8 min
- 4Tokens — the Currency of Everything7 min
- 5Why AI Output Is Probabilistic (and Why It Changes How You Build)7 min
- 6Hallucination — Why Models Make Things Up7 min
- 7What AI Is Genuinely Good and Bad At — a Capability Map7 min
- 8AI Systems vs. Traditional Software — the Mindset Shift7 min
- 9Where AI Fits in a Product (and Where It Shouldn't)7 min
- 10Responsible AI Basics for Builders7 min
Tokens — the Currency of Everything
In one line · A token is a small chunk of text (a word or part of one) — the unit that measures length and cost.
Both your bill and the context window are counted in tokens, not words. Miss this and cost and limits will surprise you.
Picture text broken into small Lego pieces. Each piece is a token. The model reads and writes in these, and you pay per piece.
'hello' is often one token, but a long Arabic word may split into several — so the same meaning can cost more tokens in Arabic.
Before text reaches the model, it passes through a tokenizer: a program that splits it into pieces from a fixed vocabulary built at training time. Common English words get one token; rare words and other languages split into smaller pieces. The model then operates on token IDs, not letters — which is why it sometimes fumbles counting characters or reversing text. Your bill is input tokens plus output tokens combined.
🧪 Try it: take an English sentence and its Arabic translation, and count each one's tokens with any tokenizer (like your model provider's tokenizer tool). Compare the counts, then try a long word and a URL and see how they split.
Common mistakes: (1) measuring length in words or characters and getting surprised when you blow past the context window; (2) thinking only input is billed — output is usually pricier per token; (3) assuming Arabic and English cost the same — the same meaning in Arabic can take double the tokens or more.
Think in tokens for any decision about cost or length: pricing a feature, chunking documents for RAG, capping user message size, or comparing models.
Don't turn everything into token obsession: for prototypes or low volume, the difference is pennies. Optimize cost when real usage justifies it.