Insights topic
Evaluation & measurement
Most enterprise AI never produces a measured outcome. These pieces cover the eval discipline, calibration data, and metrics that make results defensible.
The eval discipline gap is where projects quietly die
Enterprises will fund a pilot, ship a demo, and never build the evaluation harness that tells them whether the system is actually working on real cases. That's the eval discipline gap: a benchmark score from launch week substituting for ongoing measurement against the distribution of work the system now handles. An eval that survives production has to be built from your own edge cases, not a public leaderboard, and it has to run continuously, because model updates, prompt changes, and drifting inputs all shift performance without anyone deciding to change anything. If you can't point to the eval suite that would catch a regression, you don't have a measurement system — you have a launch announcement.
Grading the wrong exam produces confident, wrong answers
A lot of AI measurement fails not because the eval is sloppy but because it's grading the wrong exam — accuracy on a synthetic task that doesn't resemble the judgment calls the system makes in production. Automation inherits the metric it's optimized against, so if the metric rewards speed or coverage without penalizing the failure mode that actually costs money, the system will happily get faster at being wrong. This is especially visible wherever automation touches a decision with asymmetric downside — a denied claim, a missed compliance flag, a wrong price — where the average-case metric hides the tail risk that matters to the business.
Calibration data is the moat, and value isn't a single number
The organizations getting durable value from AI are the ones treating their own labeled outcomes — the corrected claims, the overridden recommendations, the escalations — as a proprietary calibration dataset, not exhaust to be discarded. That data is what lets an eval track reality instead of a static benchmark, and it compounds: the model that started at parity with a competitor's pulls ahead because it's tuned on better feedback. Equally, the value of AI isn't a single ROI number presented at a steering committee; it's a portfolio of metrics — accuracy on the cases that matter, time saved on the ones that don't, and the rate of escalation — that has to be reported honestly enough to survive scrutiny after the initial excitement fades.
The eval discipline gap: why most enterprise AI never produces measured outcomes
Companies invest tens of millions in AI capability and measure none of it rigorously. The gap between AI as theater and AI as instrument is not the technology — it is the discipline of measurement, and where it breaks down.
Read
How to build an eval that survives production
We argue constantly that you should measure the decision, not the deployment. Fair to ask in return: what does a good measurement actually look like? The constructive half — how we build an evaluation for a consequential decision.
Read
Grading the wrong exam
The number making the rounds on enterprise AI ROI is directionally true, and the executives who took it seriously and started asking harder questions did exactly the right thing. But it grades pilots as investments, when the honest function of a pilot is to produce information. By that standard, it counts learning as loss.
Read
Automation inherits the metric, not the goal
The 2026 wave of contact-center automation is real, and for most of the work it’s good. But automation doesn’t pursue your goal — it pursues your metric, relentlessly and at scale. And it removes the human agent who used to quietly bend a bad metric back toward what you actually wanted. Before you automate the contact center, be sure the number you’re about to optimize is the one you meant.
Read
The value of AI isn’t a number
The move from promise to impact was the right correction — and it stopped one step short. Value isn’t a total a company can sum; it concentrates in a few consequential decisions.
Read
Calibration data is the moat
The defensible position in enterprise AI was never the model. It’s the calibration — the accumulated record of what worked, under what conditions, after how many corrections — that a competitor can’t buy at any price. Most organizations are handing theirs to vendors without noticing it was the asset.
Read