Cost & Latency — How Your Bill Adds Up

Think of each message as a metered taxi: it charges by distance (tokens), and whatever you keep in context grows the meter every turn.

Like data roaming charged per megabyte abroad: everything you send and receive is metered, and every turn resends the same context — as if you re-downloaded the same photo over and over.

A chat that resends the whole history each turn makes you pay for the same old text again and again, so the bill piles up over a long session; summarizing old turns cuts the cost without breaking the experience.

The base equation: cost = (input tokens x input price) + (output tokens x output price) — and output is typically 3–5x pricier than input. Prices are quoted 'per million tokens'; the real bill comes from multiplying by call volume. Latency splits in two: time to first token (processing your input), then token-by-token generation — so reply length is the main driver of total time. Two essential weapons: streaming shows the reply as it generates, making the app feel far faster without changing actual latency, and prompt caching discounts a large share of repeated parts (like a long system message resent every call).

A quick calculation showing why long chats get expensive: history is resent every turn, so input grows roughly quadratically with turn count:

const IN = 0.15 / 1e6, OUT = 0.60 / 1e6;   // $/token (small model)
const system = 500, perTurn = 200, reply = 300;

let history = 0, bill = 0;
for (let turn = 1; turn <= 10; turn++) {
  const input = system + history + perTurn; // whole history resent
  bill += input * IN + reply * OUT;
  history += perTurn + reply;
}
console.log(bill.toFixed(4)); // try 10 turns vs 50 and watch the jump

🧪 Try it yourself — estimate the cost of a 10-turn chat: take real prices from a provider's pricing page and assume: a 500-token system message, 200 tokens per user message, 300 per reply. Compute (by hand or with the code above) each of the 10 turns and the total — notice turn ten costs far more than turn one even though the message is the same length. Then redo the math assuming history older than 4 turns is summarized into 150 tokens, and compare the savings. Multiply the difference by 100,000 monthly users — that's the number that gets said in the budget meeting.

• Cost-testing with short chats, then production shock. Accumulated history makes the last message of a long session cost multiples of the first — model the math on a 20-turn session, not two.
• Ignoring that output costs more. Needlessly chatty replies multiply the priciest line item — cap length with instructions and max_tokens.
• Launching without meters. Log tokens per call from day one — the first news of a cost problem should come from your dashboard, not the end-of-month invoice.

Run the cost-and-latency math before committing to any LLM feature: (expected tokens per use) x (expected monthly uses) x (price). If the number worries you, your levers in order: shorten context, summarize history, route simple parts to a smaller model, enable caching. And turn on streaming in any interactive UI — the cheapest perceived-speed win there is.

Don't let cost obsession lead before anything works: at prototype stage, output quality is the only question. And at small volume (hundreds of calls a day), cent-level differences don't justify caching and routing complexity — one engineer-hour costs more than the month's bill. Optimize when the numbers say there's something to optimize.