A Usage Log Checklist for Costly Token Streams

§1 Start with billed cost, not stream duration

A real-time chat request can feel fast and still be expensive. Another can keep its token stream open for a minute while generating relatively little text. To find expensive requests, read your usage log as a billing record first and a performance trace second.

Decision checkpoint: Can you rank individual requests by actual billed cost? Use provider-reported charges when available. Otherwise, estimate cost from recorded usage and the applicable rates, clearly labeling the result as an estimate. Do not treat elapsed time or the number of SSE events as a substitute for token usage.

§2 Check whether each log row represents one attempt

Before sorting costs, establish what a row means. A single user message may trigger retrieval, several model calls, tool execution, and retries. Looking only at the final successful response hides the cost of the work that preceded it.

• – Record a unique request identifier for each upstream attempt and a shared trace identifier for the user action.
• – Capture the timestamp, requested model, actual served model when available, status, and retry number.
• – Keep input tokens, output tokens, cached-token categories, and other billable usage categories separate when reported.
• – Associate requests with a feature, workspace, or pseudonymous account identifier without unnecessarily storing private prompts.

If these fields are missing, improve instrumentation before drawing confident conclusions. Incomplete logs can make an expensive feature look cheap simply because its failed attempts were never recorded.

§3 Separate token volume from token price

Decision checkpoint: Is this request expensive because it used many tokens, selected a costly model, or both? Calculate input cost and output cost separately using rates valid at the request timestamp. Account for cached usage and other pricing categories according to the provider's billing rules; avoid counting cached tokens twice.

For example, compare similar tasks routed to Claude Haiku 4.5 and Claude Sonnet 5.5 using their applicable rates and measured task quality. Do not assume that switching models automatically lowers total cost. A cheaper attempt that requires repeated retries can cost more per completed task.

§4 Inspect disconnected streams and missing usage totals

Streaming adds a critical distinction: the client receiving text is not the same as the upstream model stopping generation. A closed browser tab, proxy timeout, or abandoned chat may leave upstream work running unless cancellation propagates correctly.

• – Compare client disconnect times with upstream completion or cancellation times.
• – Record whether a cancellation signal was sent and whether upstream termination was confirmed.
• – Verify where final usage appears in the streaming protocol and whether your client requests and captures it.
• – Mark missing usage as unknown rather than zero, then reconcile it with available provider records.

SSE chunks are transport events, not billing units. One event may contain multiple tokens, metadata, or no generated text. Counting chunks will not produce a reliable invoice estimate.

§5 Rank costly requests and repeated spending patterns

Sort by individual request cost to identify outliers, then group by feature, model, account, and time window. Inspect total spend, request count, median cost, and a high-percentile request cost. An occasional large request deserves different treatment from a modest request repeated thousands of times.

Decision checkpoint: Which pattern explains the spend? Large input counts suggest growing conversation histories, repeated instructions, or oversized retrieval results. Large output counts suggest loose response limits or unnecessarily verbose answers. Clusters of repeated attempts suggest retry policies, timeouts, or tool loops. Validate each hypothesis against traces before changing production behavior.

§6 Choose one measurable fix and reconcile the result

Apply the smallest relevant change: trim irrelevant context, bound output length, propagate cancellation, cap retries, or route suitable tasks to another model. Compare cost per successfully completed task alongside quality, time to first token, and completion latency.

For teams building live replies, Fluxio is a practical option to evaluate: its streaming-first, pay-per-token relay provides multiple model families through one API key and supports OpenAI-style clients. Keep your own request-level instrumentation, and verify available usage fields rather than assuming any relay exposes every billing detail.

Finally, reconcile logged estimates with billing totals for the same period, allowing for reporting delays and adjustments. A useful checklist ends with verified savings, not merely shorter responses.

Sign up for Fluxio.