Measurement
Outcome metric
Definition
An outcome metric measures the result an AI system was built to change — decision quality, cost, cycle time, leakage — as opposed to an activity metric that only counts usage.
Automation inherits whatever metric it is pointed at. Aim a system at handling time and it will produce faster decisions of unexamined quality; aim it at leakage and the same underlying capability behaves differently, because the routing and thresholds get tuned to whatever is being measured. Choosing the metric is the design decision that most determines what the deployment becomes, more than the choice of model or vendor ever will.
Activity metrics are not villains, just insufficient
Usage and latency are useful operational health signals — a system nobody uses or one that times out constantly clearly has a problem worth fixing. They become harmful only when they are allowed to stand in for value in a funding conversation, where a rising usage chart gets presented as evidence of success without any accompanying claim about whether the underlying decisions actually improved. Keep activity metrics on the operations dashboard where they belong, and keep them out of the slide that argues for renewed or expanded funding.
Name the guardrail with the target
Every outcome target needs a paired constraint that must not degrade while the target improves. Cycle time down while accuracy holds. Cost down while complaint volume holds. Single-metric optimization always finds the shortcut you did not intend, because a system tuned against one number alone will happily sacrifice everything else to move it, and nobody notices until the sacrificed metric shows up somewhere expensive, like a regulatory complaint or a reserve adjustment months later.
Metrics decay if nobody revisits them
A metric chosen at launch can become stale as the underlying business changes — volumes shift, product mix changes, a new regulation adds a factor nobody was tracking before. Revisiting the outcome metric and its guardrail annually, rather than treating the original choice as permanent, catches the case where a program is still optimizing faithfully for a target that stopped mattering two years ago. This review is cheap and is frequently skipped simply because nobody owns the question of whether the original metric is still the right one.
In practice
- State one target metric and at least one guardrail metric per deployment
- Define both before the build, with the failure threshold written down
- Review guardrails at the same cadence as the target, in the same meeting
- Revisit whether the chosen metric is still the right one at least annually
