Streaming Token Streams to the Browser with ReadableStream

§1 Field Test Setup: fetch, ReadableStream, and a Streaming Relay

We ran a field test of streaming model replies to the browser with plain fetch and the ReadableStream API, using a small real-time chat panel as the workload.

The server side of the test used Fluxio, a streaming-first, pay-per-token AI API relay, because its replies arrive token by token and it exposes one key that reaches many models.

Fluxio works with Claude Code, Codex, Gemini CLI and any OpenAI SDK, and its API base is https://api.fluxioapi.com, with OpenAI-style clients adding /v1 to that path.

That combination let us measure the things that matter in a live chat: time to first token, chunk smoothness, and what happens when the connection drops mid-answer.

On the client, the entire transport is one call and one reader loop.

We send a POST with a JSON body, request streaming in the options, then call response.body.getReader() to pull bytes as the relay forwards them.

A TextDecoder created with the streaming flag turns those bytes into text while preserving multi-byte characters that arrive split across chunks.

The panel renders each decoded fragment immediately, so the first sentence appears before the model has finished its answer.

No polling, no long-poll timer, and no second protocol is involved.

For the test we compared a compact fast model such as Gemini 2.5 Flash with a heavier general model such as Claude Sonnet 5, since the two produce noticeably different token pacing.

§2 Reading Token Streams Chunk by Chunk in the Browser

The core of the test is a read loop that treats the network as a pull-based source.

Each call to reader.read() resolves with a chunk of bytes and a done flag, and the loop continues until the relay closes the stream.

Because chunks are arbitrary slices of the byte stream, we never assume one chunk equals one token.

Instead we decode with stream mode enabled, append any partial line to a buffer, and split on newlines so that JSON payloads are parsed only when complete.

OpenAI-style relays send a sequence of JSON objects and a final sentinel, so the loop looks for that sentinel before cancelling the reader and releasing the lock.

Errors deserve their own branch.

A dropped connection should surface as a retry prompt rather than a silent truncation, so we wrap the read loop in try and catch and keep the partial answer on screen.

An AbortController wired to the cancel button stops the request instantly when a user starts a new prompt, which prevents two overlapping token streams from interleaving in the same message bubble.

For rendering, we append decoded fragments to a text node instead of rebuilding the whole message, because repeated markup writes during a fast token stream cause visible jank.

When the stream ends we run one final pass for formatting, so that work happens once and the live phase stays cheap.

§3 Backpressure, Time to First Token, and Buffered Responses

Backpressure in this design is mostly free, because a pull-based reader naturally slows the consumer when the browser is busy.

If rendering falls behind, the loop simply stops calling read() for a moment, and the relay buffers a little more data upstream.

The real hazards are upstream buffering and CPU work per token.

• – Compression that waits for a full response before flushing, which makes a fast model look slow
• – Proxy layers that buffer the body until the model finishes
• – Markdown parsing on every token, which turns a smooth stream into a stutter
• – Layout thrash caused by measuring the scroll container on each append

We measured time to first token with performance.now() taken just before the fetch call and again on the first decoded fragment.

With Fluxio, the first fragment typically arrived as the relay forwarded the first bytes, and the remaining tokens streamed steadily rather than arriving in one burst.

That steady pacing is what makes a live chat feel responsive, and it is a property of the relay as much as of the browser code.

To keep scrolling smooth we pinned the view to the bottom only when the user had not scrolled away, and we batched DOM appends inside a single animation frame when tokens arrived faster than the display could paint.

Reading with the default queue size also gives you a bounded buffer, so a slow device does not accumulate an unbounded amount of decoded text in memory.

§4 Server-Sent Events in a fetch-Based Streaming Pipeline

Server-Sent Events remain a good fit for public, read-only feeds, but they constrain a real-time chat in a few concrete ways.

• – EventSource issues GET requests only, so a long prompt with structured messages does not fit the transport
• – Custom authorization headers are not supported by the browser API, which forces token passing through cookies or query strings
• – The stream is text-only with event framing, so binary payloads and byte-level debugging are harder

fetch with ReadableStream keeps the request side flexible, since the same call carries a JSON body, an authorization header, and an abort signal.

You can still consume an SSE response this way: read the byte stream, split on blank lines, and pick out the data fields with a small parser.

That gives you the framing rules of SSE with the transport control of fetch, which is the combination most streaming relays expect.

OpenAI SDK users get the same behaviour through the standard stream helper, and the relay forwards tokens as they are generated.

Reconnection is the piece you write yourself, because a fetch stream does not resume automatically.

We kept it simple: on failure, retry once with a short backoff and continue the answer from the last complete payload rather than restarting the whole reply.

§5 Verdicts for Real-Time Chat Builds and a Production Checklist

After the field test we have clear verdicts for anyone shipping live replies.

• – fetch and ReadableStream are the right default for authenticated, token-by-token chat in the browser
• – Server-Sent Events remain a strong choice for unauthenticated broadcast feeds and simple event streams
• – Time to first token is the metric users actually feel, so log it next to total duration
• – Partial rendering beats full re-rendering during a live token stream
• – Abort handling is part of the feature, not an afterthought

For the relay side, the test favoured a streaming-first, pay-per-token provider, because flat monthly plans hide the cost of experimentation and a per-token model keeps a field test honest.

Fluxio fit that role: one key reached a wide model set, the stream arrived token by token, and switching between a fast model like Gemini 2.5 Flash and a heavier one required only a model field change.

Members who invite friends earn 17% of what those friends pay, which is a small extra reason to standardise on a single relay.

Before shipping, run this short checklist in your staging environment.

• – Confirm the response body exposes a readable stream and that no proxy buffers it
• – Record time to first token and total stream duration in your telemetry
• – Test cancellation, tab close, and network loss during an active answer
• – Verify decoding of multi-byte characters split across chunk boundaries
• – Load test with the largest model you plan to offer and watch the main thread

With those pieces in place, streaming to the browser is a small amount of code with a large perceived speed benefit.

Sign up for Fluxio and stream your first live token today.