AI APIs & Structured Outputs· Lesson 5 of 7
- 1The First Call — Reaching the Model From Code7 min
- 2Streaming — The Reply Arrives Word by Word7 min
- 3Structured Outputs — JSON You Can Rely On8 min
- 4Errors & Retries — When the Call Fails9 min
- 5Rate Limits — Your Ceiling per Minute6 min
- 6API Keys & Secrets — Guarding Your Key6 min
- 7Timeouts & Idempotency — Requests That Don't Double-Fire8 min
Rate Limits — Your Ceiling per Minute
In one line · Every provider caps how many requests and tokens you can send per minute; cross it and you get a 429 and the request is rejected.
The moment your product grows or hits a spike, the rate limit is the first thing to bite you. Unprepared, your users see errors at the worst moment.
Like a bridge posted '50 cars per minute': extra cars wait in a queue. To push more through, you either pace the traffic or ask for a wider bridge.
Like the pumps at a gas station: the station can serve only so many cars an hour, and the rest wait in line. You can't force your way in — you either pace the cars entering or open more pumps.
You fire 100 requests at once but the cap is 60/min, so the rest come back 429 — and the response carries a Retry-After header telling you exactly how many seconds to wait, which you honor instead of hammering blindly; the broader fix is spreading them over time (a queue) or asking the provider to raise the limit.
For each key in each time window the provider counts two things: requests per minute (RPM) and tokens per minute (TPM) — cross either and you get a 429, usually with a Retry-After. Responses often carry headers telling you how much budget remains, so you can watch before hitting the ceiling. Client-side you add a queue that paces sends plus a concurrency limit, and longer-term you request a higher tier from the provider.
🧪 Try it yourself: you have 20 requests and a fake cap of 5 per minute — build a small queue that sends them in paced batches and honors Retry-After on any 429. Compare total time against the version that fires everything at once, and count how many requests failed.
(1) Running Promise.all over 200 items with no concurrency limit — the first burst eats the cap and everything else gets 429s. (2) Forgetting TPM: a few requests with huge prompts hit the token ceiling long before the request ceiling. (3) Treating a 429 as a bug and instantly retrying in a loop — extending your penalty instead of waiting out the Retry-After.
Any batch job or product with real traffic — build throttling before launch, because the first spike is what exposes you.
A small prototype sending one request at a time won't touch the limits — don't build a queue system prematurely. But keep the 429 handler in from day one; it's a few lines and pays off when it matters.