Streaming an LLM response can make an application feel faster because the reader sees useful text before generation finishes. It does not make the model complete sooner, and a careless implementation can cause jumping text, flickering Markdown, lost partial output, or a Stop button that fails to stop server work.
Treat streaming as a small protocol and state-machine project, not as an animation. The client must distinguish lifecycle events from text deltas, preserve incomplete data between network chunks, render at a controlled cadence, and end in an explicit completed, cancelled, or failed state.
Transport and provider statements in this article were checked against official documentation on 2026-07-24. The core sources were the OpenAI streaming guide, Anthropic streaming documentation, WHATWG Server-Sent Events specification, WHATWG Fetch Standard, WHATWG DOM Standard, WHATWG Encoding Standard, and WAI-ARIA 1.2.
Use streaming when early output is useful
Streaming is most valuable when generation is long enough for the first useful sentence to arrive materially earlier than the complete answer. Drafting, explanation, search synthesis, and conversational interfaces often fit this pattern. A short classification label, a tiny JSON object, or a result that must be validated as a whole often does not.
The main benefit is lower time to first visible output, not lower time to completion. A stream that starts quickly but stalls repeatedly may feel less reliable than a non-streamed response.
Before enabling streaming, decide what the reader can safely do with partial output. If the answer controls a payment, deletes data, changes access, or feeds another automated action, keep the partial result in a non-authoritative preview. Do not let downstream code treat a text delta as a completed instruction or valid structured object.
Keep a non-streaming path for machine-readable output that must pass schema validation, very short responses, whole-output moderation, and environments where proxies buffer incremental delivery.
Choose a transport that matches the interaction
For ordinary text generation, HTTP streaming with Server-Sent Events (SSE) is a practical default. It fits the dominant pattern: one request goes to the application server, then a sequence of events flows back. Both OpenAI’s Responses API streaming guide and Anthropic’s Messages API documentation describe SSE-based incremental responses.
SSE is a wire format as well as a browser API. The WHATWG specification defines text/event-stream, UTF-8 encoding, fields such as event and data, and a blank line as the event boundary. A server can therefore proxy a provider stream while the browser reads it with fetch(), without exposing a provider key.
Choose among three common approaches:
| Transport | Request and connection shape | Useful when | Main trade-off |
|---|---|---|---|
fetch() plus streamed body | Any supported HTTP method; one response body is read incrementally | The prompt uses POST, custom headers, authentication, or AbortSignal | The application must parse framing and define reconnection behavior |
EventSource | Browser-managed GET connection using text/event-stream | The server exposes a simple one-way event feed and automatic reconnection is helpful | Less control over request method, body, and headers |
| WebSocket | Long-lived, full-duplex connection | Both client and server must send events throughout an interactive session | More connection lifecycle and infrastructure complexity |
For most LLM text generation forms, start by evaluating fetch() with a streamed body. Use EventSource when its GET-oriented API fits the authentication model, and WebSocket when the product truly needs persistent bidirectional messages rather than incremental output alone.
Check the full delivery path. Reverse proxies, serverless platforms, compression middleware, and application frameworks may buffer small writes. Set the correct content type, avoid middleware that accumulates the body, flush headers where the runtime permits it, and send a small test event through the production-like path. A local demonstration proves only that local streaming works.
Parse events as framed data, not arbitrary chunks
A network chunk is not an LLM token, a line, or an SSE event. One event may be split across several chunks, and one chunk may contain several events. UTF-8 characters can also cross byte boundaries. Decode bytes with a streaming decoder, append text to a buffer, and extract only complete event frames.
The conceptual browser loop is:
const controller = new AbortController();
const response = await fetch("/api/generate", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ prompt }),
signal: controller.signal,
});
if (!response.ok || !response.body) {
throw new Error(`Stream failed before opening: ${response.status}`);
}
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
buffer += decoder.decode(value ?? new Uint8Array(), { stream: !done });
const frames = buffer.split(/\r?\n\r?\n/);
buffer = frames.pop() ?? "";
for (const frame of frames) {
handleSseFrame(frame);
}
if (done) break;
}This is a teaching skeleton, not a complete SSE parser. Production code must handle comments, repeated data: lines, named events, empty data, and unknown event types. Prefer a maintained SDK or SSE parser.
Normalize provider-specific events at the server boundary. OpenAI’s current Responses API guide identifies common events including response.created, response.output_text.delta, response.completed, and error. Anthropic documents a lifecycle built from message_start, content-block start/delta/stop events, message_delta, message_stop, optional ping events, and stream-level errors. Anthropic also warns that new event types may be added and clients should handle unknown types gracefully.
Normalize those shapes into an application protocol such as start, text_delta, complete, and error. Preserve provider request IDs in server logs and keep provider branching out of rendering. Never parse partial tool JSON as complete; Anthropic documents tool input as strings that must be accumulated first.
Render stable text instead of repainting every delta
The network may deliver fragments faster or less regularly than the screen should update. Appending to the DOM for every tiny delta can cause excessive layout work, noisy screen-reader announcements, and Markdown that repeatedly changes structure.
Separate the received buffer from the visible buffer. Add validated deltas to memory immediately, but batch visual updates with requestAnimationFrame or a short measured cadence. The HTML Standard defines requestAnimationFrame() as a request for a callback before the next repaint; it is a scheduling tool, not a guarantee that every network delta will be displayed.
Reserve space for the response, append instead of replacing the whole node, keep Stop fixed, and auto-scroll only while the reader remains near the bottom. Announce completion once instead of sending every fragment to an ARIA live region.
Markdown needs special handling because incomplete syntax is normal. Display escaped plain text until completion, or render only stable blocks while keeping the unfinished tail as text. In either case, sanitize generated HTML.
Respect prefers-reduced-motion and use text such as “Generating,” “Stopping,” and “Response incomplete” so status does not depend on animation. WAI-ARIA 1.2 defines status as a live-region role with an implicit polite announcement, which makes it suitable for concise lifecycle updates. Keep the rapidly changing generated text outside that status region.
Make cancellation and errors explicit states
A Stop button should abort work across the whole path: browser read, server-side provider request, and tools that can safely be cancelled. The DOM Standard defines AbortController and AbortSignal; passing a signal to fetch() stops the browser request, but the server must also close upstream generation where supported.
Cancellation is not an error. Preserve the text already received, label it Stopped, and do not present it as a complete answer. Offer Regenerate or Continue only if your application can define their semantics clearly.
Model terminal states separately: completed, cancelled, failed before output, failed after partial output, and disconnected with unknown upstream outcome. Preserve and label partial text because a retry from scratch may produce a different answer.
An HTTP 200 status only confirms that the stream opened. Anthropic documents that an error event, including an overload error, can arrive inside an already-open stream. Therefore, handle both pre-stream HTTP failures and in-stream error events. Require the provider’s explicit completion event; end-of-file alone should not silently become success.
Avoid automatic retries after partial output. Tool-using systems need stronger protection because a retry can repeat an external action. See fixing LLM tool-calling errors for that boundary and token budgets and circuit breakers for limiting runaway work.
Moderate partial output before treating it as safe
Streaming exposes text before whole-response checks can evaluate it. OpenAI’s official guide warns that partial completions are harder to moderate and states that generation-time moderation scores, when requested, arrive after the full output rather than with partial deltas. This is not a vendor-specific UI detail; it is a product-design warning.
Choose a display policy from the risk of the use case:
- Immediate display: appropriate only when partial text has been judged acceptable for the audience and context.
- Buffered display: hold a sentence, paragraph, or small time window before showing it, allowing local checks to run first.
- Gated reveal: stream to a server-side or hidden buffer, then display only after whole-output checks pass.
- No streaming: use when a partial unsafe statement would create unacceptable harm even if removed moments later.
If a later check blocks the answer, do not briefly render the text and then hide it. That leaks the content to the reader, accessibility tree, screenshots, client logs, or browser extensions. For sensitive applications, buffer before display and keep raw output out of analytics and error reporting. The checklist for reviewing AI-generated code illustrates the same principle in another high-consequence output type: generated material remains untrusted until the relevant checks pass.
Measure perceived latency and completion quality
Average request duration cannot tell you whether streaming improved the experience. Record milestones from the user’s perspective and from the server’s perspective:
Record time to request acceptance, first valid event, first visible text, explicit completion, visible cancellation, and confirmed upstream cancellation. Track inter-chunk gaps and the rate of failures after partial text appears. Together these distinguish provider startup, uneven delivery, rendering delay, interface responsiveness, and resource control.
Segment metrics by model, route, output-length band, client type, and deployment version. Use percentiles: a few long stalls may dominate frustration while barely changing the mean. Also measure rendering work, which can look like network latency on mobile devices.
A production preflight should verify:
- split multibyte characters and event boundaries decode correctly;
- several events in one chunk parse correctly;
- unknown, heartbeat, and malformed events end safely;
- cancellation propagates upstream and preserves labelled partial text;
- disconnect, timeout, overload, and mid-stream errors remain distinct;
- Markdown fragments never become unsanitized HTML;
- screen readers receive status without delta-by-delta noise;
- completion requires an explicit terminal event.
Finally, compare the streamed and non-streamed variants with the same workload. Streaming is successful when readers receive useful information sooner without more confusion, unsafe exposure, duplicate actions, or unexplained incomplete answers. If the only improvement is a typing animation, the protocol complexity is not paying for itself.
Frequently asked questions
Does streaming reduce LLM response time?
It usually reduces time to first visible output, not total generation time. Measure both first visible text and explicit completion so a faster-looking interface does not hide stalls or a slower finish.
Should I use SSE or WebSocket for LLM streaming?
Use HTTP streaming or SSE for the common one-request, one-output flow. Consider WebSocket when the application needs a persistent two-way session with client messages arriving while the server is also sending events.
Why does an SSE stream work locally but arrive all at once in production?
A proxy, platform, framework, or compression layer may be buffering small writes. Test the complete production-like path, verify Content-Type: text/event-stream, and inspect event arrival times in the client rather than assuming that a successful HTTP response proves incremental delivery.
How should I render Markdown while tokens are still arriving?
Treat the unfinished tail as unstable. Either show escaped plain text until completion or render only complete blocks and keep the tail as text; sanitize any generated HTML before insertion.
What should happen when the user clicks Stop?
Abort the browser read, propagate cancellation to the server and provider where supported, preserve the received text, and label it as incomplete. Do not convert cancellation or end-of-file into a successful completion state.
Related guides
- Reduce LLM output latency
- Design LLM timeouts, retries, and idempotency
- Plan an LLM fallback strategy
- Migrate to the Responses API
Official sources
All sources below were checked on 2026-07-24.