AI model selection is a habit, not a decision

Reaching for the frontier tier is the default in almost every serious deployment we see, and it deserves more credit than it usually gets. AI model selection at the start of a build is a decision made without the information that would settle it. You do not know the hardest input your system will meet, because the system has not met anything yet. You have a scoping document, a handful of test cases someone wrote on a Thursday, and a well-founded suspicion that users will do things nobody imagined. Against that, the largest available model is insurance. It fails less often in ways you did not anticipate. It tolerates a mediocre prompt. It survives being pointed at a task nobody scoped — which is, reliably, the task it will eventually be pointed at.

So the teams defaulting to the most capable model on offer are not being lazy, and they are not simply being sold to. They are pricing uncertainty, which is the correct response to work you cannot yet characterise. We would rarely tell a client to optimise inference cost in the first months of a deployment. Early on, the expensive failure is a system that does not work, not a system that costs more than it strictly needed to. Pay the premium. Ship the thing.

The premium is not the problem. The problem is that it outlives the uncertainty that bought it.

The default outlives the doubt

Six months into production, you hold the very thing you lacked on day one. Thousands of real requests. A distribution of what people actually ask, as opposed to what the pilot assumed. A catalogue of the hard cases, compiled at no cost by the users who found them. The uncertainty that justified the insurance has been measured away by ordinary operation — and almost nobody goes back and re-decides.

This is structural rather than negligent. A choice made under uncertainty leaves no trigger for revisiting it once the uncertainty resolves. Nothing breaks. No alert fires. The system works, it has always worked, and working is precisely what you paid for. Because the decision was not wrong when it was made, there is no moment at which it becomes visibly wrong. It simply stops being a decision and becomes a procurement habit — a line item that renews itself because no one owns the question of whether it should. We have sat in reviews where a team could describe in detail how their retrieval had changed, how their prompts had changed, how their approval steps had changed, and could not say who had chosen the model, or when, or against what.

One standard, unequal stakes

Part of what keeps the habit in place is that the question people ask has no answer. "Which model is best" is unanswerable in the abstract, and a team that keeps asking it will keep concluding, sensibly enough, that the biggest one is the safest bet. The tractable questions are different and there are two of them: what standard of accuracy does this particular decision require, and what does the gap between that standard and current performance actually cost. Only the second is measurable, and it is measurable only once you have answered the first.

Skip both and you end up applying a uniform quality standard across decisions with wildly unequal stakes. The same tier drafts an internal meeting summary and screens a customer application. One of those needs to be roughly useful; the other needs to be right nearly always, and defensible when it isn't. Value in these systems does not spread evenly across throughput — it concentrates in a few consequential decisions, and so does risk. Treating every call as though it carries the same weight means overpaying on the summary, which barely matters, and underthinking the application screen, which matters enormously. A decision that demands high assurance does not get it from a larger model. It gets it from measurement, from oversight, from a defined threshold and a way to see when you fall below it. Buying up a tier feels like an answer to that requirement while leaving it entirely untouched.

The invoice is the shallow version

There is a version of this argument that is only about money — you are spending more than you need to on inference, here is the arbitrage. That version is real, and it is the least interesting thing here. Compute prices move, and the cost you underwrote is not the cost you will pay, so an argument built on the invoice ages badly in both directions. Set it aside.

The real cost is epistemic. "We use the best model available" is an assurance story that requires no evidence to tell. It sounds like a claim about quality, it is received as a claim about quality, and it survives any meeting in which someone asks whether the system is working — because who is going to argue for using something worse. But it says nothing whatsoever about how often the system is right, on which inputs, or how anyone would know. It is a substitute for the sentence that would actually matter: we know how well this works, and here is how we know. Substitutes are dangerous in proportion to how comfortable they are, and this one is very comfortable. It is one of the quieter reasons so much enterprise AI never produces a measured outcome. The assurance question felt answered at the moment of purchase.

AI model selection needs evals

This is the practical trap, and it closes tighter than most teams realise. Routing a task to a smaller model requires two things: knowing what "right enough" means for that task, and being able to detect when you have dropped below it. Both are evaluation. An organisation that cannot measure quality on its own workload cannot make that judgment — not because it lacks discipline, but because it lacks instruments. It has no way to distinguish a smaller model doing the job adequately from a smaller model doing it badly in a way that will surface two quarters downstream in a number nobody traces back.

So the frontier default stops being a choice and becomes the only position the instrumentation permits. You are not paying a premium for capability. You are paying it for the absence of a measurement capability, permanently, with no mechanism by which that ever changes. Which is why we treat routing as a downstream benefit of an evaluation that holds up in production rather than as a cost project in its own right. Build the ability to know how well the system works and the ability to route falls out of it. Try to route without it and you are guessing with more confidence than before.

What you bought, what you never decided

Two things are worth doing at the point of deployment, and neither is a review that never comes. Grade decisions by the assurance they require before you select anything, so that the stakes are on the table while they can still shape the design. Then set an explicit revisit trigger — a date, a volume of production traffic, a moment when you will have the characterisation you lack today — and attach it to the choice, because nothing else will ever prompt you to look again.

Defaulting to the largest available model is a rational purchase of insurance against a risk you cannot yet size. That is worth paying for, and we would say so again. But insurance you never re-underwrite is not prudence, it is inertia wearing prudence's clothes. What you bought was time to find out how well your system works. What you still owe is the finding out — and buying the best model is not a decision about quality. It is the absence of one, and the most expensive way there is to avoid ever learning the answer.