Cached Input Tokens on Your Real-Time Chat Bill

§1 Two token meters behind every live reply

A streaming chat app makes an AI answer feel immediate: words appear while the model is still generating the response. Behind that smooth experience, billing usually tracks two different things: the input sent to the model and the output it generates.

Input includes your instructions, the user's message, conversation history, and any reference material attached to the request. Output is the generated answer. Tokens are small pieces of text, not necessarily whole words, and providers generally price them in large units such as one million tokens.

Cached input tokens are reusable input that qualifies for a cache-related billing category. They are not a separate kind of streamed answer, and they do not automatically mean free usage.

§2 Cached input reuses prompt processing, not old answers

Imagine a support app that sends the same lengthy product guide with every customer question. Without caching, the model may need to process that repeated material again for each request. Prompt caching lets a supported service reuse some previously processed input.

The customer can still ask a new question and receive a newly generated answer. This is different from saving a finished response and displaying it again. Input caching reuses work on the prompt, rather than substituting an old answer for a fresh one.

Eligibility depends on the provider and model. Some systems cache eligible prefixes automatically; others require explicit configuration. Minimum lengths, expiration windows, and cache creation charges can also differ. Never assume that identical text guarantees a discounted cache hit.

§3 Reading cached tokens without counting them twice

A usage report might show 10,000 total input tokens, including 8,000 cached tokens, plus 1,000 output tokens. In that reporting format, the cached tokens are a subset of input, not another 8,000 tokens to add on top.

For a simplified example, suppose uncached input costs $2 per million tokens, cached input costs $0.20 per million, and output costs $8 per million. These are illustrative rates, not a quote for any service.

• – The 2,000 uncached input tokens cost $0.004.
• – The 8,000 cached input tokens cost $0.0016.
• – The 1,000 output tokens cost $0.008.
• – The total request cost is $0.0136, excluding any additional applicable charges.

Without the cached-input discount, the same token counts would cost $0.028 at those example rates. Your actual calculation must follow the provider's usage definitions, especially if cache creation and cache reads have separate prices.

§4 Streaming changes delivery, not cache eligibility

Streaming sends output incrementally, often through server-sent events, or SSE. It does not inherently make input cacheable. The same caching rules may apply whether the answer arrives all at once or token by token.

Caching can reduce input-processing work and sometimes improve time to the first token. It does not guarantee faster generation afterward. Likewise, a response arriving in tiny chunks does not mean each chunk creates a separate billable request.

When logging streamed usage, check whether your API returns usage in a final event and whether your client requests that event when required. A disconnected browser does not necessarily cancel upstream generation, so connect cancellation handling to the actual API request.

§5 Making repeated chat context easier to reuse

Place stable instructions and shared reference material before changing user content when the provider uses prefix-based caching. Avoid putting a fresh timestamp or random identifier at the beginning of every prompt, because early changes can prevent reuse of the later prefix.

Measure cache hits instead of assuming they happen. Track total input, cached input, output, and billed cost together. Also compare similar workloads: a longer conversation may cost more overall even when its cache-hit rate improves.

§6 Checking cached usage through Fluxio

Fluxio is a streaming-first, pay-per-token AI API relay that fits apps needing live replies through one API key. Its OpenAI-style interface can simplify integration with compatible SDKs while you evaluate model options.

Before budgeting, verify the selected model's cache support, current rates, and how cached usage appears in Fluxio's responses or billing records. A shared interface does not mean every underlying provider has identical caching rules.

Sign up for Fluxio to build your streaming app.