Streaming — The Reply Arrives Word by Word

In one line · Instead of waiting for the whole reply to finish, streaming sends it to you token by token as it's generated — that ChatGPT-style typing feel.

Glossary:InferenceLLMSSEStreamingToken

Why it matters

Users feel a product is far faster when text appears immediately, even if the total time is the same. It transforms the experience of any chat.

It's live broadcast versus downloading a whole video: instead of staring at a spinner until everything's ready, you see the first sentence while the rest is still coming.

An analogy

Like a simultaneous interpreter: they don't wait for the whole speech to end before translating — they hand you word after word as it comes, so you follow from the first second.

You set stream: true, so instead of one response you get a series of SSE events, each a data: line carrying a small text delta — you concatenate and render them as they arrive until a [DONE] line signals the end.

How it works

You add stream: true to the request, so instead of one JSON blob the connection stays open and the server sends SSE events: each data: line carries a small chunk with a text delta inside. You read the body as a stream, split it into lines, parse each chunk, and append the deltas in the UI as they arrive. When the [DONE] line shows up you close and treat the reply as complete — any drop before it means what you have is partial.

const res = await fetch(url, {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ model: "model-name", stream: true, messages }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let text = "";
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  for (const line of decoder.decode(value).split("\n")) {
    if (!line.startsWith("data: ") || line.includes("[DONE]")) continue;
    const delta = JSON.parse(line.slice(6)).choices[0].delta.content ?? "";
    text += delta; // render incrementally here
  }
}

Practice

🧪 Try it yourself: set stream: true and render the reply word by word in a terminal or simple page until [DONE] arrives. Then cut the connection mid-stream (with AbortController) and see how your code handles the partial reply.

Common mistakes

(1) Buffering the whole stream into a variable and rendering at the end — you paid streaming's complexity with zero user benefit. (2) Not handling a mid-stream drop: the connection can die halfway, and storing that fragment as a complete reply corrupts your data. (3) Calling JSON.parse on the whole body — a stream is separate SSE lines, not one JSON document.

When to use it

Any chat or long reply shown to a user waiting at the screen — an early first word changes how the whole product feels.

When NOT to use it

Background batch jobs, or replies your code parses whole such as structured JSON — there non-streaming is far simpler and nobody is watching anyway.

LET'S BUILD ITDone reading — now build it yourself.Render a streamed reply word by word

Test your understanding

1. Does streaming make the reply finish faster?

2. The connection dropped mid-stream and your code saved the received text as the final reply. What's the right behavior?

3. You set `stream: true` then call `res.json()` as usual. What happens?

Related concepts

Errors & Retries — When the Call Fails The First Call — Reaching the Model From Code Timeouts & Idempotency — Requests That Don't Double-Fire

Was this clear?