Live AI Token Streams Through nginx Without Buffering

§1 Why SSE tokens arrive in batches through nginx

A live AI reply should appear as it is generated, not suddenly arrive as a finished paragraph. If your application streams correctly but users see long pauses followed by large chunks, nginx may be buffering the response.

Server-Sent Events, or SSE, sends a response that stays open while the server writes small messages. AI applications commonly use this format to deliver generated text incrementally. nginx sits between the client and your application, so its response-handling settings can affect when those messages become visible.

The goal is simple: let nginx pass incoming bytes downstream promptly instead of collecting them first.

§2 The SSE response your application must send

Start with the application, because nginx cannot turn a completed response into a live stream. Your endpoint must send the Content-Type header with the value text/event-stream, write events as they become available, and flush output when its framework requires it.

A basic event contains a line such as data: hello followed by two newline characters. That blank line finishes the event. Without it, an SSE client may wait even when bytes have already arrived.

• – Send Cache-Control: no-cache to discourage caching of the live response.
• – Consider sending X-Accel-Buffering: no so nginx can disable response buffering for this upstream response, unless its configuration ignores that header.
• – Avoid middleware that collects the entire response before sending it, including some compression layers.

An SSE event does not necessarily equal one model token. Providers and SDKs may group text differently, and network layers can combine small writes.

§3 A minimal nginx location for unbuffered SSE

Place the following location block inside the appropriate server block. This example forwards /chat/stream to an application listening on port 3000 on the same machine. The application must handle that same path.

location /chat/stream { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_buffering off; proxy_cache off; gzip off; proxy_read_timeout 300s; }

proxy_buffering off is the central setting. It tells nginx to pass the upstream response along as it receives it rather than buffering it for later delivery. You do not need to tune proxy buffer sizes to accomplish this.

proxy_http_version 1.1 explicitly selects HTTP/1.1 for the upstream connection. Clearing the Connection header avoids forwarding nginx's default close value. This does not force browser connections to use HTTP/1.1; clients can still connect through HTTP/2.

proxy_cache off disables proxy caching for this location, while gzip off disables nginx's gzip response filter there. Neither setting removes buffering or compression inside your application or another intermediary.

proxy_read_timeout 300s allows up to five minutes between successive reads from the upstream. It is not a five-minute maximum stream duration. Choose a value that fits expected pauses, and consider application-generated SSE comment heartbeats during idle periods.

§4 Testing SSE before and after the proxy

Validate the configuration with nginx -t, then reload nginx using your deployment's normal procedure. Test the application directly before testing the public route:

$ curl -N -i http://127.0.0.1:3000/chat/stream

Next, run the same test against your nginx endpoint. The -N option disables curl's own output buffering. Include the method, authentication, and request body your endpoint actually requires.

If the direct connection streams but the proxied connection stalls, inspect nginx and any CDN or load balancer in front of it. If both stall, investigate application flushing, middleware, and event boundaries first. proxy_request_buffering controls incoming request bodies, not streamed responses.

§5 Keeping Fluxio token streams live end to end

Fluxio fits applications that need a streaming-first, pay-per-token AI API relay. One key can support models such as Claude Haiku 4.5 and Gemini 2.5 Flash through an OpenAI-style client. Use https://api.fluxioapi.com/v1 as the base URL for those clients and enable streaming in the request.

Keep the API key on your backend. Read Fluxio's stream incrementally, forward events without accumulating the complete reply, and stop upstream work when the client disconnects. nginx can preserve live delivery only if every other layer does too.

Sign up for Fluxio.