Model Routing for Live Token Streams: A Field Test

§1 Field Test Setup for Routing Live Chat Turns Across a Streaming Relay

Model routing means choosing a model for every chat turn instead of committing one model to an entire conversation. In a live product, that choice shows up to users as the moment the first token appears on screen, so we tested routing where it is most visible, inside a real-time chat loop. Over several weeks we routed more than four thousand production turns through one streaming relay and recorded time to first token, streamed tokens per second, task completion, and cost per resolved turn. The relay exposed a single API key across a wide catalog of models, which meant a routing decision changed nothing in our credentials or client code, only the model field on the request. We used an OpenAI-style client pointed at the relay base URL with streaming enabled, so every reply arrived token by token and we could watch each routing choice land live. Traffic came from four workloads: short support questions, multi-step reasoning, code refactoring requests, and long-context document summarization. Before each turn, a lightweight classifier looked only at prompt shape, conversation length, and the failure flags from the previous turn, then the routing table sent the turn to either a cheap fast model or a heavier reasoning model. Two models carried most of the traffic: Claude Haiku 4.5 for the low-latency tier and Gemini 3.1 Pro for turns that needed deeper reasoning.

§2 Time to First Token as the Routing Metric That Actually Moves

Static benchmark scores describe a model in isolation, while time to first token describes a product in use. In our field test the fast tier answered about seventy percent of turns and kept median time to first token under one second, whereas the heavy tier on the same relay connection reached first token closer to two and a half seconds. Users feel that gap even when the final answer is identical, because a streaming interface trains people to expect motion the instant they press enter. Routing must therefore trade the cost of waiting against the cost of being wrong, and that trade is different for every turn shape.

• – Sending greeting, formatting, and clarification prompts to the fast tier so the cursor starts moving immediately
• – Reserving the reasoning tier for turns that carry multi-step logic, arithmetic, or long dependency chains
• – Escalating a turn to the heavy tier only after the fast tier flagged low confidence in its first streamed tokens
• – Measuring time to first token per route rather than averaging it across the whole conversation
• – Keeping a warm connection to the relay so transport setup never masquerades as model latency

Once we routed on time to first token instead of perceived model prestige, the fast tier absorbed more traffic than expected without any measurable drop in resolution rate. The relay's token-by-token delivery made this measurement trustworthy, because we could timestamp the first streamed token rather than guessing when a full response buffer flushed.

§3 Turn-Level Accuracy Tests That Justify Escalation

The second question was whether routing to a cheaper model actually hurt quality, so we ran blind evaluations per turn type rather than per conversation. Short factual and operational prompts resolved identically on both tiers, which means paying for a reasoning model on those turns is pure overhead. Multi-step reasoning and long-document summarization were the categories where escalation paid for itself, since the heavy tier produced fewer mid-answer corrections and far less user follow-up. Gemini 3.1 Pro was the escalation target in this test and it earned that slot on structured reasoning, while Claude Haiku 4.5 held its ground on conversational turns where tone and speed mattered more than depth.

• – Measuring resolution rate per turn category instead of blended accuracy across mixed traffic
• – Counting user retries and follow-up clarification turns as accuracy failures in disguise
• – Tracking cost per resolved turn so a cheaper model does not win on price while losing on repetition
• – Logging the router's confidence score alongside the verdict to calibrate escalation thresholds
• – Testing long-context summarization separately because token volume alone does not predict difficulty

The verdict from this round is that escalation should be narrow and evidence-based. A router that sends everything to the heavy model wastes budget, and a router that sends everything to the fast model pays for it in retries. The winning policy escalated roughly three in ten turns, and almost all of those escalations came from the reasoning and summarization categories.

§4 Fallback Rules and Stream Continuity When a Routed Turn Fails

Routing adds a new failure surface, because a turn can fail after the token stream has already begun. In a real-time chat, a half-written answer that dies mid-stream feels worse than a slow answer, so fallback logic has to respect the live reply rather than restart it blindly. Our rule set kept a short prefix buffer of the first streamed tokens, checked for truncation and malformed output, and only retried when the stream had not yet produced visible content. When a retry was allowed, we rerouted to a different tier and streamed the replacement immediately so the interface never went silent.

• – Buffering the first few streamed tokens so a fallback cannot duplicate visible text already shown to the user
• – Retrying silently only before the first visible token appears in the chat surface
• – Switching tiers on retry instead of hammering the same model during a provider-side slowdown
• – Preserving server-sent event semantics so event type and finish reason stay valid across a reroute
• – Surfacing a short recovery message when a stream dies past the point where a clean retry is possible

Because the relay speaks an OpenAI-style streaming contract and delivers tokens continuously, the fallback code stayed small and transport-agnostic. We did not have to write per-vendor stream parsers for each model in the catalog, which is exactly the kind of plumbing a routing layer should not contain.

§5 Routing Policy Verdicts From the Field Test

Three verdicts came out of the runs, and they hold up across all four workloads. First, route on turn shape and measured latency, not on model reputation, because most turns never use the depth a flagship model offers. Second, escalate on evidence such as low confidence, long context, or multi-step structure, and keep escalation near thirty percent of traffic to stay cost-effective. Third, treat stream continuity as a first-class requirement, since a clean token-by-token reply matters more to perceived quality than a small accuracy delta.

For the relay itself, our verdict is straightforward: a streaming-first, pay-per-token API relay is the right substrate for this design. One key reached the full model catalog including Claude Haiku 4.5 and Gemini 3.1 Pro, replies arrived token by token over an OpenAI-compatible interface at the base URL with the version path appended, and our existing OpenAI SDK code ran unchanged against it. The same relay also worked with command-line coding agents, so the routing policy we validated in chat carried over to developer tooling without rewrites. Teams that want one routing layer across many models, with streaming behavior they can trust and billing that follows actual token use, should evaluate Fluxio at fluxioapi.com, with the API base at api.fluxioapi.com for OpenAI-style clients.

Sign up for Fluxio and route your first live chat turn today.