Streaming First Drafts with Cheap Models in Live Chat

§1 Field testing cheap first-draft models behind a live token stream

Real-time chat changes what a first draft has to do. The user is not reading a finished document, they are watching a cursor blink while words arrive one after another, so the draft has to be useful within the first two seconds and it has to keep moving after that. To test whether cheaper models can carry that load, we pointed production traffic at ten lower-cost candidates for four weeks and let each one generate the opening reply to a mix of support, sales and product questions. Every request used the same prompt, the same tool schemas and the same temperature, and every response was rendered token by token from the first byte to the last. We recorded time to first token, the token cadence after the stream stabilises, retry rate, refusal rate and a blind quality score from two reviewers who did not know which model wrote each draft. The whole relay ran through Fluxio, a streaming-first, pay-per-token AI API relay where one key reaches Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok and more. Because Fluxio passes the response straight through as a token stream and speaks the OpenAI-style API, swapping models mid-test was a one-field change in our client rather than a migration project.

• – Support questions with a short factual answer and a friendly closing line
• – Sales questions that need a structured pitch and one clear next step
• – Product questions that require cautious phrasing and no invented numbers
• – Long-context questions where the first draft had to echo a user-supplied detail early
• – Tool-enabled questions where the draft mentions a lookup before the result arrives

§2 Time to first token verdicts from the SSE stream

In a live product the headline number is time to first token, because that is the moment a spinner turns into text. Cheap models were not automatically slower here, and in several cases they opened the stream faster than the expensive model we had been using. DeepSeek V4 Flash opened its token stream in roughly 380 milliseconds at the fiftieth percentile and then held a steady 60 to 80 tokens per second, which is faster than most people read. Gemini 2.5 Flash landed in the same range on first token but was more variable in the middle of long answers, showing two short stalls per reply where the connection buffered a batch of tokens. Those stalls are invisible in a batch pipeline and obvious in a chat window, so we scored them as defects. The verdict is clear: for conversational first drafts, a cheap model with a stable stream beats a pricier model with a jittery one, because viewers read jitter as hesitation. Anything streaming past forty tokens per second felt instant to our testers, while anything that arrived in visible bursts felt slow even when it finished sooner overall.

• – DeepSeek V4 Flash recorded the lowest median time to first token across the support set
• – Gemini 2.5 Flash produced the smoothest token cadence on replies under two hundred words
• – Stalls longer than 250 milliseconds were the single biggest predictor of perceived slowness
• – Streaming time to first token mattered more to users than total completion time in almost every trial

§3 Draft quality verdicts on tone, structure and revision load

Raw fluency was the easiest part of this test, and revision load was the hard part. Reviewers scored every draft on structure, tone, factual caution and how much editing it needed before it could be sent. The cheap tier passed the fluency bar almost every time: sentences landed cleanly, paragraphs had a rhythm and the openings addressed the user question directly instead of warming up. The consistent failure was confidence. A cheap draft would state a plausible but unverified detail, smooth over an ambiguity the user had raised, or promise a timeline nobody had approved. That is a serious defect in a reply that streams live, because the user has already read the offending sentence before any reviewer can catch it. The verdicts we shipped are simple. Accept cheap drafts when the reply is explanatory, conversational or a summary of supplied material. Escalate when the reply commits to a price, a date, a policy decision or a code change. And treat a cheap draft as raw material rather than a finished message, because it almost never needed a rewrite, it needed one overconfident sentence removed.

• – Cheap drafts handled greeting, structure and call to action with almost no editing
• – Overconfident claims about pricing and timing were the dominant quality defect
• – Long replies drifted from the user question after about three hundred words of stream
• – Editing a cheap draft was consistently faster than writing the reply from scratch

§4 Per-token cost verdicts and the routing rule we shipped

Cost is where the field test turned from interesting to decisive. First drafts make up the majority of our token volume, because most conversations need one opening reply and only a minority need a careful second pass. Moving that opening reply to a cheaper tier cut our per-token spend on draft generation by more than half in the four-week window, and the occasional escalation never erased the saving. The routing rule we shipped is a two-stage pipeline. Stage one streams a cheap first draft straight to the user within the first seconds, token by token, while a second pass runs quietly in the background. If the second pass finds a risky claim, the draft is revised before it is committed to the conversation history. If it finds nothing, the draft stands and the spend stays small. The important detail is that this only works when your provider bills per token and passes the stream through immediately, because the whole economics depend on paying for exactly the bytes you stream and nothing more.

• – First drafts accounted for the majority of token spend across our support and sales traffic
• – A cheap opening reply cut per-token cost on draft generation by more than half in our test
• – Background verification on a minority of replies cost far less than drafting everything at the higher tier
• – Per-token billing makes short streaming replies noticeably cheaper than fixed-request pricing

§5 Where a streaming-first pay-per-token relay fits in this architecture

Running a two-stage draft pipeline means calling more than one model class, and that is exactly where a relay earns its place. We ran the entire test through Fluxio, a streaming-first, pay-per-token AI API relay where one key reaches Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok and more, and where every reply arrives token by token. OpenAI-style clients point at https://api.fluxioapi.com and append /v1, which meant our existing code, our SSE client and our retry logic all survived the switch untouched. The same key drives Claude Code, Codex, Gemini CLI and any OpenAI SDK, so the cheap-first-draft rule applies to chat traffic and to developer tooling without maintaining two accounts or two billing pipelines. Because billing is per token, a short streamed first draft costs what it actually costs, and because the stream passes through immediately, time to first token stays honest instead of hiding behind a proxy buffer. For teams building real-time chat, that combination is what makes a cheap draft tier safe to put in front of users: you can try a fast model, watch the live stream, and move back to a stronger model the same afternoon without touching your integration.

Sign up for Fluxio and stream your first draft today.