Streaming AI Retries Without Double-Billing Users

A live AI reply stops halfway through a sentence. Your app retries, the answer continues, and everything looks fine. Behind the scenes, however, two requests may have generated tokens. Without careful bookkeeping, your user could pay twice for what felt like one conversation turn.

The safest approach is to separate three things: the user’s request, each attempt to fulfill it, and the final charge. Network retries may happen several times. Charging should follow one explicit policy, not the number of times your code runs.

§1 One user action needs one stable request ID

When someone presses Send, create a unique ID for that logical request. Keep it unchanged across retries caused by timeouts, dropped connections, or repeated button presses while the request is pending. A deliberately new message gets a new ID.

This ID becomes an idempotency key: a label that lets your server recognize repeated submissions as the same operation. Scope it to the authenticated account, and store a fingerprint of the request content. Reject reuse with different content rather than silently treating a changed prompt as an old request.

If your AI provider supports idempotency, follow its documented rules. Do not assume that sending an arbitrary header prevents duplicate generation or charges.

§2 A billing record must survive simultaneous retries

Before starting generation, save a durable record containing the account, request ID, status, and billing state. Enforce uniqueness in the database so two servers cannot independently create the same logical job.

• – A pending request should return its existing status rather than launch another generation.
• – A completed request should return its saved result without creating another charge.
• – A request with an uncertain outcome should enter recovery instead of immediately starting over.

For billing, use a unique ledger entry tied to the logical request. If you reserve credits first, record the reservation and settle or release it once. Make the final charge and billing-state update atomic, or use a payment system’s documented idempotency mechanism when crossing system boundaries.

§3 A broken token stream does not prove generation stopped

Server-sent events, often called SSE, deliver replies incrementally. If that connection breaks, the upstream model may still be generating tokens. Retrying immediately can create a second billable generation even though your app stopped receiving the first.

Record each upstream attempt separately, including its provider request ID when available. After a disconnect, check the original attempt’s status if the provider offers that capability. Cancellation can help limit further work, but it does not prove that no tokens were generated or charged.

For stronger recovery, let your backend consume the upstream stream and save events before forwarding them. The browser can reconnect to your backend and replay saved events. Sequence numbers help avoid displaying the same text twice. Resuming an interrupted upstream stream requires provider support; SSE alone does not guarantee it.

§4 Retry only failures that are safe to repeat

Temporary rate limits and service errors may justify retries. Invalid credentials and malformed requests generally do not. Respect retry guidance, use increasing delays with randomness, and cap both attempts and total waiting time.

A timeout is an unknown outcome, not proof of failure. Without upstream idempotency or status lookup, you cannot guarantee that a replacement attempt avoids another provider charge. You can still guarantee one charge in your own ledger and absorb duplicate upstream costs under a clearly stated policy.

§5 Streaming relays still need your billing safeguards

Fluxio is a useful option for streaming AI apps: it is a streaming-first, pay-per-token API relay with one key and OpenAI-style client compatibility. For example, you could offer Claude Haiku 4.5 while keeping your application’s request IDs and billing ledger consistent.

Check documented retry, cancellation, and usage-reporting behavior before implementing recovery. Do not assume relay compatibility includes billing deduplication. Test disconnects, duplicate submissions, and server crashes, then verify that each logical request produces at most one customer charge.

Sign up for Fluxio to build your streaming AI app.