Prompt Caching: Budgeting a Streaming Chat

§1 Step 1: Separate Streaming From Prompt Caching

Before forecasting savings, distinguish two mechanisms. Streaming delivers generated content incrementally. Prompt caching reuses previously processed prompt prefixes, reducing repeated input processing. It does not replay a saved answer: the model still generates a fresh response, and output tokens remain a separate budget item. Caching can shorten the wait before generation starts without changing how output tokens are generated. ([developers.openai.com](https://developers.openai.com/api/docs/guides/prompt-caching?trk=article-ssr-frontend-pulse_x-social-details_comments-action_comment-text&utm_source=openai))

Build your budget around four categories: ordinary input, cache writes, cache reads, and generated output. Treat streaming as the delivery experience, not an automatic discount.

§2 Step 2: Put Reusable Instructions Before Changing Messages

Arrange the request so stable instructions and reference material precede changing conversation content. Prefix caching depends on matching the beginning of a request; inserting changing content near the front can prevent reuse farther downstream. Minimum cacheable lengths and supported controls vary by model. ([developers.openai.com](https://developers.openai.com/api/docs/guides/prompt-caching?trk=article-ssr-frontend-pulse_x-social-details_comments-action_comment-text&utm_source=openai))

• – Keep system instructions and tool definitions consistent between requests.
• – Place timestamps and request-specific details after reusable material wherever practical.
• – Append conversation turns instead of rewriting earlier messages unnecessarily.

For a trial with Claude Haiku 4.5, check the selected route's caching requirements before implementing controls. Anthropic supports automatic caching and explicit cache breakpoints, but an intermediary's compatibility must be verified separately. ([platform.claude.com](https://platform.claude.com/docs/en/build-with-claude/prompt-caching))

§3 Step 3: Calculate Cold and Warm Request Costs

Use a deliberately hypothetical rate card, not a provider quote: ordinary input costs $2 per million tokens, cache writes cost $2.50, cache reads cost $0.20, and output costs $8. Assume each request contains an unchanged 8,000-token prefix, 1,000 additional input tokens, and 500 output tokens.

Request cost equals ordinary-input cost plus cache-write cost plus cache-read cost plus output cost. Assign each input token to its applicable billing category rather than counting cached tokens twice.

• – Without caching, each request costs $0.018 for input plus $0.004 for output, totaling $0.022.
• – A cold request costs $0.020 to write the prefix, $0.002 for additional input, and $0.004 for output, totaling $0.026.
• – A warm request costs $0.0016 to read the prefix, $0.002 for additional input, and $0.004 for output, totaling $0.0076.

Ten identical-size requests, comprising one write and nine hits, therefore cost $0.0944 instead of $0.22: approximately 57% less overall. Notice that the cold request costs more, and output spending does not fall. This simplified example excludes growing history, retries, and other fees.

§4 Step 4: Measure Cache Reuse Throughout the Stream

Instrument actual usage rather than inferring savings from response speed. Anthropic reports cache creation and cache read token counts separately from ordinary input. Its ordinary input field excludes those cached categories, so summing the three gives total input. ([platform.claude.com](https://platform.claude.com/docs/en/build-with-claude/prompt-caching))

For OpenAI Chat Completions streaming, requesting stream_options.include_usage produces an additional usage chunk before completion. An interrupted stream may never deliver that chunk. Record missing usage explicitly and reconcile it with billing records rather than assigning zero cost. Verify equivalent behavior through your relay. ([developers.openai.com](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create))

§5 Step 5: Stress-Test Expiration and Conversation Growth

Run cold, immediate-repeat, and delayed-repeat tests. Anthropic's default five-minute cache lifetime starts when the request begins, so a long streaming response consumes part of the reuse window. A longer lifetime is available, but its pricing must be evaluated against your traffic pattern. ([platform.claude.com](https://platform.claude.com/docs/en/build-with-claude/prompt-caching))

Track cost per completed conversation, cached tokens as a share of total input, and time to first token. Include a no-hit budget scenario. Do not mistake cheaper repeated history for permission to retain unlimited history or generate unnecessarily long answers.

§6 Step 6: Validate Caching Through Fluxio

Fluxio fits teams seeking a streaming-first, pay-per-token relay with one key across multiple providers and OpenAI SDK compatibility. Before relying on caching savings, verify that your chosen route forwards caching controls, exposes usage categories, and applies the expected billing rates. SDK compatibility alone should not be your acceptance test.

Compare cold and warm requests through the same route, then scale only after measured costs match your forecast.

Sign up for Fluxio.