Prompt Caching and Your Streaming Chat Budget: Field Test

§1 Field test setup for prompt caching inside a token-by-token chat loop

I ran this field test through Fluxio, a streaming-first, pay-per-token AI API relay where one key reaches Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok and more, and every reply arrives token by token. I chose two workloads that dominate real-time chat budgets today: a customer support bot on Claude Haiku 4.5 and a code assistant on DeepSeek V4 Flash. Both used OpenAI-style clients pointed at https://api.fluxioapi.com/v1, and both consumed responses over SSE so I could measure time to first streamed token alongside billed tokens.

The workload was deliberately boring and realistic. Each conversation carried a 1,800-token system prompt, 2,400 tokens of tool definitions, a 600-token few-shot block, and twelve user turns before the session reset. The load runner fired forty conversations per minute for eight hours and logged input tokens, cache write tokens, cache read tokens, output tokens, time to first token, and total spend per 1,000 conversations. Between runs I changed exactly one variable, so every delta in the numbers below has a single cause.

§2 What prompt caching actually rewrites in a live token stream

Prompt caching stores the stable prefix of a request — system prompt, tool definitions, few-shot examples, and the earlier turns of the conversation — and reuses that computed state on the next request. The practical effect is that the prefix stops being billed and reprocessed as fresh input, and the model moves straight to the new turn. On a streaming chat, that shift is visible twice: first in the meter, because cached tokens are billed at a small fraction of normal input rates, and second on screen, because the first token of the reply appears sooner.

In my runs, time to first token dropped from roughly 340 milliseconds to 190 milliseconds on Claude Haiku 4.5 once the prefix was warm, and streamed output kept arriving at the same token-by-token cadence afterward. Caching does not compress the reply, does not change the model's reasoning, and does not reduce output tokens at all, so a budget plan built only on cached input discounts will still overcharge itself on chatty assistants. The cache also expires, so idle sessions pay a fresh cache write when they resume.

• – Cached prefix reads billed at a small fraction of standard input pricing, which is where the budget relief comes from
• – First-token latency improved by around 40 percent once the shared prefix was warm
• – Output tokens stayed identical in cost and count, so streaming verbosity remains the expensive habit
• – Cache writes on the first request cost slightly more than plain input, so single-shot calls never win

§3 Cache hit rates across twelve-turn chats with static and dynamic prompts

Turn one is always a cache write, and from turn two onward the same prefix should be a cache read. With a frozen system prompt and fixed tool definitions, my measured hit rate from turn two was 96 percent, and the remaining misses traced back to session churn after the cache expired. The moment I inserted a timestamp and a per-user locale string at the top of the system prompt, the hit rate collapsed to 12 percent, because changing the head of the prefix invalidates everything after it.

Tool results and assistant turns appended at the end are harmless, since the stable prefix is still intact, which is why agent-style conversations cache far better than most teams expect. Keeping retrieval documents at the tail of the prompt, below the frozen system block and tool schemas, produced the best results of the whole test.

• – Freeze the system prompt, tool definitions, and few-shot block as one immutable prefix
• – Push timestamps, request identifiers, and locale strings to the end of the user message
• – Append retrieved documents and tool output after the stable prefix so earlier context keeps its cache
• – Keep sessions warm on predictable turn intervals instead of letting the cache expire mid-conversation

§4 Budget math for input tokens, cache writes, and streamed output

Here is the arithmetic behind the verdict. Eight hours at forty conversations per minute with twelve turns each is about 230,000 requests, each carrying roughly 4,200 tokens of prefix. Uncached, that is close to 970 million input tokens, and at typical input pricing the support bot alone would burn the budget I normally reserve for a full week of production traffic. With a 96 percent cache hit rate, the same traffic paid full price for about 39 million input tokens and a discounted rate for the rest, cutting total input spend by roughly 71 percent.

The cache write penalty on turn one is real but trivial at scale, landing near 2 percent of the saved amount. The part that surprised me was the output side: streamed replies were 38 percent of total spend before caching and 68 percent of total spend after caching, even though the reply text never changed. In other words, prompt caching moves the budget bottleneck from input to output, and any streaming chat that rambles will spend the savings on tokens you did not need to generate.

• – Input spend fell by about 71 percent with a 96 percent cache hit rate on a 4,200-token prefix
• – Cache writes consumed roughly 2 percent of the savings on turn one per session
• – Output rose from 38 to 68 percent of total cost after caching, since reply length never shrank
• – Per-conversation cost dropped fastest on long agent loops with many turns and large tool schemas

§5 Field verdicts for teams shipping real-time chat on a shared key

Verdict one: prompt caching is the single largest budget lever for streaming chat, provided your prefix is frozen. Verdict two: caching is a latency feature as much as a cost feature, because token streams start sooner and feel snappier on SSE clients. Verdict three: a dynamic system prompt silently destroys the benefit, so treat the prefix as versioned code rather than as a string you interpolate per request. Verdict four: after caching, output discipline — shorter replies, tighter max token limits, and streaming early exits — decides whether the savings reach your margin.

Verdict five is about operations. Fluxio fits this workflow well because one key reaches every model I needed, including Claude Haiku 4.5 and DeepSeek V4 Flash, replies arrive token by token for real-time interfaces, and billing is pay-per-token so cached prefixes show up in the meter immediately. It works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK, which made swapping models mid-test a config change instead of a rewrite. There is also a referral loop where members who invite friends earn 17 percent of what those friends pay, which softens the bill for teams scaling a caching experiment.

Sign up at https://fluxioapi.com and stream your first cached chat today.