kylebeyke.com

applied AI systems and technical analysis

AI Token Costs: The Hidden Incentive Problem

[2026-04-20] [Kyle Beyke]

Table of contents

AI token costs are not a side issue in modern AI systems. They are one of the clearest places where provider incentives and user incentives can diverge. Most inference vendors bill by input and output tokens, sometimes with separate charges for cached input, reasoning-related usage, grounded search calls, or other add-ons. That means more tokens often mean more revenue for the seller, while the buyer usually wants the opposite: fewer tokens, lower latency, tighter outputs, and more predictable costs. This is not proof of bad intent. It is a structural incentive problem, and teams that ignore it usually pay for it later.

The key mistake is treating token volume as if it were neutral. It is not neutral. In a usage-based market, the billing unit shapes product behavior. If a provider is paid per token, it has an obvious commercial reason to make high-token use cases easier to build, easier to justify, and easier to normalize. Meanwhile, the customer’s actual goal is rarely “consume more tokens.” The real goal is to complete a task accurately, quickly, and cheaply enough to make the application worth running. That difference matters in product design, in architecture, and in procurement.

Why AI token costs create a real incentive mismatch

The mismatch starts with the billing model itself. OpenAI, Anthropic, and Google all publish token-based pricing for major API offerings, and Google’s current Gemini pricing explicitly varies by prompt size, with higher input and output prices once prompts exceed 200,000 tokens on some models. Google also prices context caching and some grounding features separately. In other words, the market is not just charging for “AI”; it is charging for how much text, context, and generated output flows through the system.

That does not mean providers are secretly trying to waste your money on every call. It does mean they are naturally rewarded when products, frameworks, and best practices expand token consumption. A provider can still improve model quality, safety, and developer experience in good faith while operating inside a model that benefits from higher usage. Those two facts can both be true at the same time.

This is also why buyers should be skeptical of any architecture advice that treats longer prompts, larger context windows, heavier agent loops, and more verbose outputs as the default path to quality. Sometimes those choices are justified. Often they are just expensive habits with weak measurement behind them. The right question is not whether a system can consume more context. The right question is whether that extra context measurably improves the outcome enough to justify the cost and latency. Research on long-context behavior has repeatedly shown that simply expanding context does not guarantee better use of the information inside it.

How providers and products can nudge you toward higher AI token costs

1. Bigger context windows can normalize waste

Large context windows are useful. They make many workflows possible. But they also make it easier for teams to stop curating inputs. Instead of filtering documents, summarizing history, or retrieving only relevant passages, many applications just send everything. That inflates input tokens immediately. It also tends to increase output tokens because the model has more material to synthesize and more room to wander. Google’s pricing pages explicitly distinguish cost tiers based on prompt size in several Gemini offerings, which means oversized prompts can directly move a request into a more expensive band.

The deeper issue is that long context is often marketed as capacity, while buyers experience it as convenience. Convenience is real, but convenience is not efficiency. The “Lost in the Middle” findings are still relevant here: models with long contexts do not reliably use all parts of that context equally well. More tokens in the window can still mean worse cost efficiency if relevance selection is poor.

2. Verbose defaults raise both latency and spend

Many AI products default to long answers because long answers feel impressive in a demo. They look thoughtful. They look premium. But long answers cost more and often reduce usability. OpenAI’s own guidance on controlling response length states that shorter responses help manage cost, improve latency, and avoid overly long or verbose outputs. That is an important admission from a first-party source: more text is not automatically better product behavior.

This is where the incentive mismatch becomes practical. A provider may sell you on quality, but the model interface may still default toward elaboration unless you explicitly constrain it. If your application does not set output caps, response schemas, stop sequences, or concise instructions, you are effectively giving the model permission to spend your budget for style points.

Reasoning-capable models can improve performance on hard tasks, but they can also consume tokens you do not directly see in the final answer. OpenAI’s reasoning documentation shows usage objects that break out reasoning tokens and warns that you can incur costs for input and reasoning tokens even before visible output is produced if the response runs into token limits. OpenAI also recommends reserving substantial token space when experimenting with these models. That is valuable capability, but it changes the economics. The user may think they are paying for a short answer, while the system is billing for a much larger internal process.

That does not make reasoning models a bad choice. It does mean buyers need to distinguish between tasks that truly need deeper reasoning and tasks that would be better served by a cheaper, faster model plus tighter prompting or structured retrieval. Anthropic’s latency guidance similarly advises choosing the model that fits the use case rather than reaching for the most capable option by default.

4. Tool chatter and agent loops can explode token volume

Every tool call, tool result, retry, and intermediate planning step can add tokens. In agentic systems, token growth is often multiplicative rather than linear because the model keeps re-reading history, tool schemas, instructions, and prior outputs. The application team may think of the workflow as “one task,” but the bill reflects many model turns. This is one reason agent demos often look elegant while production bills look ugly. The cost is in the orchestration, not just the answer.

Providers are not solely responsible for that waste. Framework defaults, permissive loop settings, oversized tool definitions, and weak stopping logic are major contributors. But the market does reward systems that make those patterns easy to adopt. More autonomy often means more token traffic unless it is tightly governed.

5. “Just send the whole conversation” is the lazy tax

Many chat applications keep replaying full history on every turn. Sometimes that is necessary. Often it is not. Anthropic’s prompt caching docs explicitly call out long multi-turn conversations, repetitive tasks, and large amounts of context as ideal candidates for caching. OpenAI says prompt caching can reduce input token costs by up to 90 percent and latency by up to 80 percent for repetitive prompt prefixes. Those features exist because replaying the same large prefix over and over is expensive.

The hidden problem is that many teams treat chat history as free memory. It is not. If the same prefix, policy block, or large document is being resent repeatedly without caching or summarization, the application is paying a tax for architectural laziness.

What practices tend to maximize AI token costs

The fastest way to increase AI token costs is to combine several expensive habits at once: long system prompts, full history replay, high-output defaults, multiple candidates, repeated retries, and broad tool schemas. OpenAI’s guidance notes that multiple completions can be requested in chat completions, and its response-length guidance recommends explicit caps and stop sequences for cost control. If you generate more candidates than you actually use, you are buying unused text.

Another common cost amplifier is oversized retrieval. Teams frequently dump entire documents into a prompt instead of retrieving only the smallest relevant chunks. They do this because a large context window makes it technically possible. But technical possibility is not economic discipline. When a request carries unnecessary tokens, the application pays twice: once on the way in and often again in a more diffuse, wordier output.

A third cost maximizer is imprecise output specification. If you want a classification, ask for a classification. If you want structured fields, use structured output. Google’s Gemini docs explicitly support JSON Schema-based structured outputs for predictable, type-safe results. That is more than a convenience feature. It is often a cost-control feature because it narrows the space of possible outputs and reduces the chance that the model will generate a long, decorative explanation when you only needed machine-readable data.

The same logic applies to retries. If your prompts are vague, your schemas are absent, and your evaluation rules are weak, your system will miss more often. Misses trigger retries. Retries multiply token usage. That is why prompt quality and output control matter economically, not just stylistically. Anthropic’s prompt-engineering guidance for business performance makes this point directly: inefficient inputs and outputs at scale become costly, and better prompting reduces unnecessary back-and-forth.

How to reduce AI token costs without hurting results

Choose the smallest model that reliably does the job

This is the highest-leverage cost decision in most stacks. Anthropic’s latency guidance says one of the most straightforward ways to reduce latency is to select the appropriate model for the use case. That advice also maps to cost. Larger or more reasoning-heavy models are valuable when the task needs them. They are wasteful when a smaller model can achieve the same business outcome.

The discipline here is simple: benchmark tasks by business result, not by model prestige. If a smaller model can classify, extract, rewrite, or summarize well enough, stop paying a premium for unnecessary capability. This is also where small-model or hybrid strategies can materially change unit economics in production.

Count tokens before you send them

Google’s CountTokens API exists for a reason: you should know what you are about to spend before you make the call. Google says the CountTokens API can calculate input tokens before inference and help estimate potential cost. Token counting should not be an afterthought in production systems. It should be part of request validation, logging, and budget policy.

This is especially important when prompts include retrieved context, long instructions, or dynamically assembled tool definitions. Token growth is often accidental. Teams discover it only after the bill arrives. A preflight token count turns that surprise into a controllable engineering variable.

Stop sending repeated prefixes from scratch

Prompt caching is one of the clearest examples of how to cut AI token costs without sacrificing quality. OpenAI says prompt caching can reduce input token costs by up to 90 percent and latency by up to 80 percent. Anthropic documents caching for prompts with many examples, large background context, repetitive tasks, and long multi-turn conversations. If your app reuses large instruction blocks or stable reference material, caching is not optional hygiene. It is core cost control.

Caching is not the only answer. You should also summarize old history, store durable state outside the prompt when possible, and avoid replaying irrelevant turns. But if your architecture depends on repeated prefixes, not using caching is usually self-inflicted waste.

Cap outputs aggressively and specify structure

OpenAI’s official guidance recommends max output controls, clear instructions, and stop sequences. Google’s structured output support lets you define response schemas so the model returns predictable data. These controls matter because unconstrained output is a budget leak. Applications should default to the shortest output that still completes the task. Long-form explanation should be earned, not assumed.

A practical rule works well here: default to terse machine-readable output for internal workflows, and expand only when the user explicitly requests more detail. That keeps token budgets aligned with actual user value.

Retrieve less, but retrieve better

Most RAG waste is retrieval waste. The fix is not “never use context.” The fix is to retrieve smaller, better-ranked chunks; deduplicate them; and send only what the model needs. Long-context support is useful, but it should not become an excuse to skip relevance engineering. The more selective your retrieval layer is, the lower your AI token costs tend to be.

This is also why document preprocessing matters. Summaries, chunking strategy, canonical snippets, and metadata filters are not just quality improvements. They are token-discipline mechanisms.

Use asynchronous discounts where latency is not user-facing

If the task does not require an immediate response, batch it. OpenAI’s Batch API offers 50 percent lower costs for asynchronous request groups over a 24-hour window. That is a direct economic win for offline enrichment, backfills, labeling, classification, and other non-interactive jobs. Many teams keep paying real-time prices for work that has no real-time requirement.

The principle generalizes beyond one vendor: do not buy premium latency for jobs that do not need premium latency. Separate synchronous product paths from background processing paths and price them differently.

Measure outcome per token, not just price per million

Price-per-million-token comparisons are useful, but they are incomplete. Different models tokenize content differently, structure outputs differently, and vary in how many retries they need for a given task. The metric that matters in production is useful outcome per token, per second, and per dollar. That means you should track task success, latency, retries, average input length, average output length, and total spend together.

A cheaper model that needs more retries may cost more in the end. A more expensive model that produces concise structured output on the first try may be cheaper at the workflow level. Token discipline is not about chasing the lowest list price. It is about measuring the whole unit economics of the task.

The buyer’s playbook for keeping AI token costs under control

The first rule is governance. Every production AI feature should have token budgets, logging, and alerting. If a workflow has no cap on input size, no output ceiling, and no visibility into retries, then the team does not really control spend. It only observes spend after the fact. Google’s token-counting guidance and OpenAI’s output controls make clear that these are controllable variables, not mysteries.

The second rule is architecture. Keep durable memory out of the prompt when possible. Cache reusable prefixes. Summarize stale history. Use structured output instead of prose when the task is structured. Split high-speed user-facing flows from cheaper asynchronous flows. Escalate to larger models only when smaller models fail a measured threshold.

The third rule is skepticism. When a provider, framework, or consultant recommends more context, more steps, more autonomy, or more reasoning, ask one question: what is the measured gain per additional token? If nobody can answer that, the recommendation is probably architecture theater.

The bottom line is simple. AI token costs are not just a finance concern. They are a systems-design concern and a procurement concern. Providers are usually paid when token usage rises. Users usually win when useful outcomes arrive with fewer tokens, less latency, and tighter control. The teams that treat that mismatch honestly will build better AI systems and spend far less doing it.

FAQ Section

Why do AI token costs create a provider-user conflict?

Because most providers bill by input and output tokens. More tokens generally mean more revenue for the provider, while users usually want fewer tokens, lower latency, and lower cost for the same task.

Are providers intentionally trying to waste tokens?

That cannot be stated as a general fact. The stronger and better-supported claim is that token-based billing creates a structural incentive mismatch. Providers can act in good faith and still benefit financially from usage patterns that are not optimal for customers.

What is the fastest way to reduce AI token costs?

Usually: pick a smaller model when it is good enough, cap outputs, remove unnecessary context, use structured output, and enable prompt caching for repeated prefixes.

Do large context windows always improve results?

No. They increase capacity, but research shows models do not always use long context reliably, especially when relevant information is buried in the middle of large inputs.

How should teams evaluate model cost in practice?

Track useful outcome per token, per second, and per dollar. That means measuring success rate, retries, latency, input size, output size, and total workflow cost together instead of relying only on list price.

Sources

      Sign up for the kylebeyke.com newsletter and get notifications about my latest writings and projects.