Cached Input Tokens on Streaming AI Bills: A Field Test

§1 Setting Up a Replayed Prompt Set for the Cached Token Field Test

Streaming chat products send the same conversation prefix again on every turn, which makes input caching one of the few billing levers that grows in value as a session gets longer. To test it honestly, we built a replay harness that fires the same 40-turn session over a token stream, with a stable system prompt of about 3,200 tokens and tool definitions that never change between calls. Each request opened a server-sent events stream, waited for the first visible token, then collected the usage block that arrived with the final chunk. The harness ran through Fluxio, a streaming-first pay-per-token relay where one key reaches providers such as Claude Sonnet 5.5 and Gemini 3.1 Pro, and where replies arrive token by token to any OpenAI SDK. Because the relay exposes an OpenAI-style base at https://api.fluxioapi.com with a /v1 suffix for OpenAI clients, the same measurement script ran unchanged across providers. We recorded cached input tokens, billed input tokens, cache write tokens, and wall-clock time to first token for every call. The point of the replay set was to isolate billing behavior from prompt drift, so nothing in the prefix was rewritten between turns.

§2 Reading Cache Writes and Cache Reads in the Live Token Stream

Input caching splits into two ledger lines that behave very differently under streaming. A cache write stores the prefix so the next call can reuse it, and it is usually billed at a modest premium over ordinary input. A cache read reuses that stored prefix, and it is typically billed at a steep discount, sometimes near an order of magnitude cheaper than fresh input. The catch is that these numbers show up in the usage object attached to the stream, not in the tokens you see scrolling on screen. In our SSE runs, the visible token stream looked identical whether the prefix was cached or freshly parsed, so the only way to know what happened was to read cached input tokens from the final chunk and compare them against prompt tokens. A healthy session showed a large cached input count on turns two and beyond, a small cache write count when a prefix branch was created, and a billed input total that stayed flat as the context grew. A broken session showed zeros in the cached field, which meant every turn paid full price for a prefix it had already seen. Reading that field is the difference between trusting the stream and auditing it.

§3 Numbers from the First Hundred Streaming Requests

Across 100 streamed requests on a 40-turn session, the results settled into a clear pattern. Turn one always paid for the full prefix and triggered a cache write. Turns two through forty read that cached prefix, and the billed input on those turns landed around 12 to 18 percent of what the same input would have cost uncached. On a session with roughly 9,000 input tokens at turn forty, uncached billing would have charged for all 9,000 every turn, while the cached run charged for the growing delta and read the rest at the discounted rate. The measured savings climbed steadily with session length, because the proportion of fresh context was higher early on and the cached fraction dominated later turns. Wall-clock behavior mattered too. Cache hits reduced time to first token noticeably on long prefixes, since the provider skipped re-parsing the stored context and could start emitting the reply sooner. The relay kept those savings visible, because the usage block arrived with the closing stream event instead of hiding behind a batch job. For a real-time chat product, that means the billing picture and the user experience move in the same direction, since cheaper input also means a faster first token.

§4 Where Cached Input Tokens Stop Cutting the Bill

Caching is not a universal discount, and the field test exposed four places where it quietly fails. First, any edit near the top of the prefix invalidates everything after it, so a system prompt that embeds a timestamp forces a full cache write on every request. Second, very short sessions never amortize the cache write premium, and calls with under a few hundred prefix tokens often cost more with caching than without it. Third, the discount only applies to input, so streaming output tokens are billed at full price and a verbose reply can erase the input savings in a single turn. Fourth, routing a session across providers breaks the cache, because the stored prefix lives inside one provider cache and does not follow the conversation when you switch models for a fallback. We saw this last case directly when a session moved from Claude Sonnet 5.5 to DeepSeek V4 Flash after a rate-limit response and paid the cache write again on the new provider. The operational fix is to keep prefixes stable, pin a session to one model when possible, and treat output length as the budget line that dominates the bill on long streaming replies.

§5 Verdicts on Cached Input Token Billing for Real-Time Chat Apps

Our verdict on cached input tokens is that they are the single highest-leverage billing control for streaming chat workloads, provided the product is built to exploit them.

• – Keep the system prompt and tool schema byte-stable across turns so cache reads remain valid
• – Log cached input tokens beside time to first token on every closing stream chunk
• – Pin each session to one model so a provider cache is not recreated during fallbacks
• – Budget output tokens as the dominant line item on long streaming replies

Read the cached input field on every closing stream chunk and treat a silent zero as a billing defect that users will never report. Budget for output as the dominant cost on long replies, and treat cache reads as the discount that makes deep context affordable rather than as a reason to add more context. If you want the same setup without maintaining per-provider plumbing, Fluxio fits cleanly: one key, pay-per-token billing, token-by-token streams that work with Claude Code, Codex, Gemini CLI and any OpenAI SDK, and a relay where cached input tokens sit in the usage data your client already parses. The referral program adds a separate income stream, since members who invite friends earn 17 percent of what those friends pay. For a real-time chat bill that shrinks as sessions grow, sign up at Fluxio and run this replay against your own prefixes.