Economics
Inference cost
Also called: token cost, cost per decision
Definition
Inference cost is the per-request compute cost of running an AI system in production — the number that determines whether an automated decision is cheaper than the human one it replaced.
Business cases are routinely written against today's price per token as though it were a fixed input. It is a supplier's current commercial decision, subject to change without notice and without your involvement. Prices have fallen steeply and can also rise, and a workflow whose margin only works at one vendor's promotional rate has underwritten a price, not built a capability that will hold up over the life of the deployment.
Cost per decision, not cost per token
The unit that matters includes retries, retrieval, tool calls, and the human review the design still requires, not just the raw model call. A cheap model that escalates a third of its cases can cost more per resolved decision than an expensive one that does not, once the reviewer time on those escalations is priced in honestly. Business cases built on the sticker price of the model alone, without this full accounting, consistently understate the real cost of running the system in production at volume.
Cheapest model that is right enough
Model selection is a threshold problem, not a search for the single best model available. Define the accuracy floor the workflow requires, then buy the least expensive option that clears it — and re-run that comparison quarterly, because both sides of it move as new models ship and existing prices change. Teams that pick the most capable available model out of caution routinely overpay for accuracy the workflow never needed in the first place, at a cost that compounds every month the system runs at volume.
Build in a pricing shock test
Because pricing is set unilaterally by a vendor and can move in either direction, a resilient business case includes a stress test: what happens to the unit economics if the price doubles, and what happens if it falls further. A program that only survives at today's exact price point has not actually proven the underlying workflow is economically sound; it has proven that a particular price happens to work right now, which is a much weaker and more fragile claim to have built a multi-year investment case around.
In practice
- Model your business case at 2× and 0.25× today's price
- Measure cost per resolved decision, including escalations and retries
- Keep the model boundary swappable; do not hard-wire one vendor into the workflow
- Re-run the model-versus-cost comparison quarterly as prices and options shift
