Token Variability and Credit Rate Design for LLM-Powered Features

A flat credit rate mis-prices a meaningful share of LLM requests before a single customer ever logs in, because token consumption varies by design, not by accident. The variance is structural, built into how language models tokenize text, generate output, and charge for the two differently.

Tokens are subword units, not words and not characters, and in English text you get roughly four characters per token. A user who asks a product to "summarize this document" has no intuitive way to estimate what that action costs, because the mapping between a plain-language request and its token count depends on vocabulary, formatting, and the specific tokenizer the model uses. Guessing by feel does not work here, and treating it as if it does is how credit rates fail.

The asymmetry between input and output tokens compounds the problem. Output tokens must be generated one at a time, in sequence, which makes them computationally more expensive to produce than input tokens, which a model can process in parallel. Across major providers, output tokens cost roughly three to eight times as much as input tokens, with a median ratio near 4-to-1. A credit rate built around a flat token count will overcharge short, input-heavy queries and undercharge long, verbose completions, and it will do so consistently, every time a customer's usage leans one way or the other.

Tokenization itself is not standardized across providers, so the same input sequence can produce roughly 140 tokens under one model and more than 180 under another. If a product routes traffic across multiple models or switches vendors, cost precision degrades, because the token count underlying a single credit is no longer a fixed quantity.

Newer models add a third category: reasoning tokens, representing internal "thinking" steps that never appear in the user-facing response but still get billed, typically at output-token rates or higher. Any feature built on chain-of-thought or multi-step reasoning carries this cost whether or not the customer ever sees it. Hidden overhead stacks on top of all of this: retry logic, retrieval augmentation, context window management, and embedding generation each consume billable tokens that most teams never track at the prompt level, and the true cost of a feature is routinely higher than its published per-request estimate suggests.

How input/output asymmetry shapes the credit rate problem

Token variability has a specific shape, and the input/output ratio is the dimension a credit designer has to account for first. Output tokens cost three to eight times more than input tokens, so whether a flat credit rate turns a profit or a loss depends on feature type, not just usage volume. Different features occupy very different points on that spectrum, and a rate built for one will misprice the others.

A search assistant that retrieves and feeds in large chunks of source material sits at the input-heavy end: it sends large retrieved chunks into the model and gets back a short answer. Its cost is dominated by cheap input tokens, so a flat credit rate calibrated to a blended average will overcharge this feature relative to its actual cost of goods sold. A documentation generator or content-creation tool sits at the opposite end: it sends a modest prompt and produces a long completion, so its cost is dominated by expensive output tokens, and that same flat rate will undercharge it, compressing or inverting the vendor's margin on every use.

Conversational agents complicate the picture further, because their cost profile shifts over time as a function of feature type and conversation length. Each turn in a multi-turn exchange appends the prior conversation history to the input, so the input token count grows turn over turn even when the user's own message stays short. A conversation that starts cheap can become expensive purely as a function of its own length, independent of what the user is actually asking.

The practical consequence of applying one credit rate across feature types is cross-subsidization: terse, input-heavy features end up subsidizing verbose, output-heavy ones. That arrangement can hold for a while, but it turns into a margin problem the moment usage mix shifts toward the expensive side, whether because customers adopt the output-heavy feature more heavily or because a conversational feature's average session length grows.

Model-tier price spreads and the multi-rate problem

Model choice adds a second axis of variability on top of the input/output asymmetry, and the two compound each other. The scale of the problem is visible in the 2026 model price table: GPT-5.2 and GPT-5.2 Pro carry an output-to-input ratio near 8-to-1, while DeepSeek V3.2 carries a ratio only a small fraction of that ($0.28 per million input tokens against $0.42 per million output tokens, by one pricing source, with OpenRouter's listing of $0.2088 and $0.3096 landing at a nearly identical ratio). The asymmetry established in the previous section is itself model-dependent, so a credit rate tuned to one model's spread will misprice every request served by a model with a different spread.

GPT-5.2 Pro costs vastly more than GPT-4o Mini for the same workload. If a customer's account was priced against the budget model but draws on the premium model, it generates a loss on every request, and that loss accumulates silently until someone reconciles infrastructure spend against credit revenue.

Products that expose model choice to customers, or that route requests automatically based on task complexity, face a compounding version of this problem: the routing logic itself decides whether each credit is over- or underpriced. A product that adds a cheaper or more capable model without revisiting its credit rate will shift its margin position without anyone deciding to make that trade, because the same number of credits now buys a different amount of actual compute than it did when the rate was set.

A reasonable objection holds that most products do not expose model choice to customers, so the spread should not matter. Automatic routing by task type is itself a form of multi-model deployment, even when the customer never picks a model directly, and any future model upgrade or substitution forces a recalculation of the credit rate regardless of whether routing is manual or automatic. Locking a credit rate to the assumption of a single, unchanging model is a brittle foundation for a pricing system meant to last longer than one model generation.

What credit-based pricing assumes

Credit-based pricing has become the standard answer to this variability because it solves a real problem for customers, converting a pile of heterogeneous token costs into a single unit people can reason about. Instead of tracking input tokens, output tokens, model tiers, and reasoning tokens separately, a customer buys credits and spends them, and the abstraction holds up as long as the exchange rate underneath it is set correctly. It solves the vendor's margin problem only on that condition.

The trade embedded in this design is a transfer of forecasting risk from customer to vendor. Once credits are sold at a fixed rate, any later increase in infrastructure cost, or any shift in usage toward more expensive features, compresses margin on credits that have already been sold and cannot be repriced retroactively. That risk has an accounting dimension too: under ASC 606, prepaid credits are recorded on the books as a deferred revenue liability until the customer actually consumes them. The cash arrives before the revenue does, so a credit rate that underprices the real cost of delivery creates a liability that costs more to fulfill than it brought in. Most revenue automation infrastructure was built for seat-based SaaS and only later adapted to handle usage-based consumption; the credit accounting problem is a finance infrastructure issue as much as a pricing design issue. Systems built to count seats were never built to count tokens.

Two recent cases show what happens at both ends of this calibration problem. GitHub moved Copilot to token-based AI Credits on June 1, 2026: GitHub's own documentation states that it prices consumption per model and converts it into credits at a published rate of 1 AI credit equal to $0.01 USD. When metered billing took effect, developers publicly reported bill shock, because single requests ate up double-digit percentages of a monthly allowance. Even a well-resourced engineering organization can set an exchange rate that feels punishing to the people paying it.

Gong took a different structural approach. Gong announced Gong Credits in June 2026, following customer notification emails sent in late May 2026, building a company-wide shared credit pool on top of its existing per-seat pricing. The credit layer meters AI Trackers, MCP server access, and API-based workflows, but the human-facing core of the product stays inside seat pricing. Credits can absorb the variable AI cost layer without forcing a repricing of the stable, predictable seat layer underneath it. Both cases are credit-based pricing decisions. One produced customer-facing sticker shock; the other kept the variable layer contained.

The four pricing archetypes and the usage profiles each one can absorb

Four pricing models currently compete for LLM-powered features, and each allocates token-cost variability differently between vendor and customer. Picking among them is a decision about which party is better positioned to absorb the dominant source of variance in a given product's usage profile, not a decision about which one looks best on a pricing page.

Per-token pricing passes all variability straight to the customer, who pays what the model costs, with no vendor abstraction between usage and invoice. It fits API-first, developer-facing products, where customers know enough to monitor and optimize their own consumption. Pure token pricing tends toward a race to the bottom unless real value sits on top of it, because as frontier model prices fall, the per-token price has to fall with them, creating constant downward pressure with no floor underneath. The forecasting cost lands on the customer: one bad batch job and a monthly bill looks nothing like the one before it, and that kind of invoice surprise raises both support load and churn risk for the vendor.

Credit packs work as a prepaid wallet. The vendor absorbs rate risk, the customer gets cost predictability and pays upfront, and the upfront cash gives the vendor a strong working-capital position. The entire design problem rests on the exchange rate: set it wrong in one direction and the vendor loses margin on every redemption, set it wrong in the other and customers churn at the sticker shock. This model fits self-serve and SMB customers who want a known spend ceiling and have no reliable way to forecast their own token usage. The Cursor incident from June 2025 shows what happens without a ceiling: a single developer in a single month ran up an enormous invoice through uncapped usage inside an annually billed plan. Spend caps and usage alerts belong in the product as engineering requirements, not as optional interface polish.

Per-action or outcome-based pricing charges for a completed task instead of for compute consumed, so it absorbs all token variability inside the vendor's margin and prices the product on delivered value. The model is value-aligned and definitionally hard to execute: what exactly counts as a completed action? An AI agent that attempts a task and partially succeeds raises a dispute the contract needs to resolve in advance. Without a precise, auditable definition of what counts as complete, outcome billing invites exactly the kind of dispute it was meant to avoid, and it requires metering infrastructure robust enough to record outcome events, not just token counts, reliably at scale.

Hybrid pricing splits the variability by layer: a base subscription gives the vendor predictable revenue, and a metered usage layer passes incremental variability to the customer above that floor. Gong Credits is the live example here, with seat pricing covering the human-facing core of the product and credits metering the AI workload layer separately, because the two layers carry different variability profiles and deserve different pricing logic. Hybrid pricing is also the most operationally complex of the four: it requires metering the usage layer, rating it against a credit exchange rate, and reconciling both against a separate subscription invoice. For enterprise AI, it has become something close to a structural default, because the subscription gives enterprise buyers the procurement path their purchasing process requires, while the usage meter captures AI value that a flat seat count cannot.

Setting a credit exchange rate across input/output mix, model tier, and hidden overhead

A defensible credit exchange rate is a weighted rate, built from the actual usage mix the product expects to see, with explicit buffers for the overhead layers that never appear on a model provider's published price sheet.

Start by weighting the rate to the actual feature mix. If a product combines a retrieval-based search assistant with a long-form content generator, it needs to weight its rate by the actual share of traffic each feature drives, because the input-heavy and output-heavy sides of that mix pull the true cost in opposite directions. Setting the rate against the output-heavy feature alone will overcharge the input-heavy traffic; setting it against the input-heavy feature alone will quietly erase margin on every output-heavy request.

Model tier has to enter the calculation directly, not as an afterthought. A rate built for GPT-4o Mini-level costs will lose money the moment a request routes to GPT-5.2 Pro, because that model carries a roughly 8-to-1 output-to-input ratio, far above the much smaller ratio on DeepSeek V3.2. Any product that routes across models, whether by customer choice or automatic task-complexity routing, needs a rate that reflects the blended cost of that routing logic as it actually runs in production, not as it ran on the day the rate was first set.

Hidden overhead needs its own line in the calculation. Reasoning tokens, retry logic, retrieval augmentation, context window management, and embedding generation all add billable tokens that do not appear in a feature's advertised prompt-and-response cost. A rate that only accounts for the visible request and ignores the overhead layers it depends on will underprice every feature that uses those mechanisms, and the gap grows as a product adds more agentic or multi-step features over time.

Pega's announcement at PegaWorld on June 8, 2026, that Pega Infinity 26, available in the third quarter, would eliminate the AI token tax through a flat fee per resolved case, points to where some vendors are trying to take this problem next: fixing the unit of value instead. That approach still has to solve the same underlying problem this piece has traced throughout. Whatever unit a vendor charges against, be it a token, a credit, or a resolved case, the rate behind it has to reflect the real, weighted cost of serving the traffic that will actually hit it, across input and output tokens, across model tiers, and across the overhead that never makes it onto the invoice.