Retry Token Streams Without Double-Billing Users

§1 Where Duplicate Charges Appear in SSE Retry Loops

Streaming replies make retries tempting and dangerous at the same time. When an SSE connection drops halfway through a token stream, the client sees silence, assumes failure, and fires the request again. The provider may already have generated hundreds of tokens and may be billing for them at that moment. If your usage ledger records every attempt as a separate event, the user pays twice for one answer. This bug rarely shows up in request-and-response testing because the happy path completes cleanly. It appears under real network pressure, on mobile connections, during provider failover, and at the exact second your retry middleware decides to be helpful.

The root cause is almost never the retry itself. It is the missing identity for the logical operation. Each HTTP attempt carries a fresh request identifier, so your invoice service treats the attempts as independent billable work. Fix identity first, and retries become a performance feature instead of a billing defect.

• – client-side SDK retries fired after an SSE stream closes unexpectedly
• – server-side gateway failover to another upstream while tokens are still generating
• – a user pressing regenerate while the first reply is still streaming
• – background workers replaying failed usage webhook deliveries
• – load balancer timeouts where the upstream keeps generating after the client is gone

§2 Assign One Idempotency Key Before the First Token

Generate a single idempotency key the moment the user submits the prompt, before any network call leaves your service. That key is the identity of the logical request, and it must survive every retry, every reconnect, and every automatic model fallback. Attach it as a header on each attempt, and persist it with the prompt hash, the user identifier, and the requested model tier in a row that enforces uniqueness. The first attempt to claim the key marks the row pending and owns the billing outcome; later attempts carrying the same key must never call the model again as new work.

Instead, a replaying attempt should receive the cached result or, for live token streams, a re-play of buffered events so the client still sees a smooth token-by-token reply. Keep the idempotency window longer than your worst-case client retry horizon, and align it with provider usage finalization so a late usage report can still match the original key.

• – a UUID generated server-side at prompt submission time
• – an unchanged header passed through every SDK retry wrapper and gateway
• – a persisted pending row with a unique index and a timestamped expiry
• – a replay buffer storing partial SSE events for reconnecting clients
• – a stable key surviving model fallback and provider failover

§3 Deduplicate at the Billing Boundary, Not in the Client

Client-side guards help, but they cannot be trusted for money. A user with two browser tabs, a mobile app that restores its session, or a retrying service worker will create attempts your client code never saw. Billing correctness has to live where charges are recorded. Every usage event should carry the idempotency key and a monotonic sequence number, and your ledger should accept each key exactly once, no matter how many attempts reached the network. The first event wins, and every later event is either merged into it or discarded.

Treat the provider usage report as authoritative for token counts and your idempotency key as authoritative for uniqueness. When two reports arrive for one key, keep the earliest and reconcile the later one against it, never sum both. Meter accruals only after generation finishes or the stream is declared dead, so a partial attempt cannot be billed twice for the same output.

• – a unique constraint on idempotency key inside the usage events table
• – a monotonic sequence number per token stream so partial writes merge safely
• – a reconciliation job matching provider usage reports to your own token counts
• – a compensating credit entry issued when a duplicate charge slips through
• – an audit trail linking every invoice line to one logical prompt submission

§4 Retry Policy for Live Token Streams and Interrupted SSE

Streaming changes the retry calculus because partial output has real value. If the SSE stream dies after three hundred tokens, do not blindly re-run the full prompt and bill it as new work. Classify the failure first. A reset before the first token means a safe full retry under the same idempotency key. A reset after tokens arrived means a stream resume, a cached replay, or a retry that is explicitly credited back to the user.

Back off exponentially with jitter so reconnecting clients do not stampede a rate-limited provider, and always honor Retry-After signals. Cap total attempts and total elapsed time, then fail visibly with a clear error instead of quietly burning budget in the background.

• – a full retry with the same idempotency key when no tokens were delivered
• – stream resume or cached replay once the first attempt produced output
• – exponential backoff with jitter capped at thirty seconds between attempts
• – respect for provider Retry-After headers during rate limiting
• – a circuit breaker that fails fast instead of stacking billable requests
• – an attempt counter that stops before user patience or budget is exhausted

§5 Ship the Whole Flow in Five Minutes with a Streaming Relay

A streaming-first, pay-per-token relay removes most of this plumbing. Fluxio gives you one key that reaches Claude Sonnet 5.5 and GPT-6 Astra among many others, with replies arriving token by token. It works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK by pointing the base URL at api.fluxioapi.com and appending /v1 for OpenAI-style clients. Because metering is pay-per-token, your idempotency layer maps cleanly onto one billable logical request rather than a pile of attempts. Members who invite friends also earn 17 percent of what those friends pay.

Wire the safety layer first, then stream. This checklist is the five-minute version.

• – create one relay key and keep it in a secret manager rather than the client bundle
• – generate an idempotency key at prompt submission and send it on every attempt
• – point an OpenAI SDK client at api.fluxioapi.com/v1 and stream a token-by-token reply
• – buffer SSE events per idempotency key so reconnecting clients resume without a second charge
• – write one usage event per key and verify it against the provider usage report
• – classify failures: full retry before the first token, replay after it
• – enable backoff with jitter and a hard maximum attempt count before production traffic

Sign up for Fluxio and stream your first token today.