Measurement
Baseline (and counterfactual)
Definition
A baseline is the alternative an AI system is measured against — ideally the cheapest unglamorous approach that mostly works, not the absence of any system at all.
Measured against zero, every project succeeds. Measured against the existing rule set, the spreadsheet, or the experienced person who has done the work for nine years, many do not. The counterfactual — what would have happened without this system — is the part that cannot be observed directly and has to be approximated deliberately through holdouts or staged rollout, which takes planning that most projects skip because it feels like it slows down the launch.
Pick a baseline that can beat you
If the comparison cannot lose, it is not a measurement. The discomfort of choosing a strong baseline before you build is exactly the discomfort that makes the eventual result credible to a CFO, a board, or a skeptical operations lead who has seen prior AI pilots overstate their impact. A weak baseline produces a flattering number that nobody outside the project trusts, which defeats the entire purpose of measuring in the first place, since the point of the exercise was to produce a number worth acting on.
Approximating the counterfactual
Holdout groups, staggered rollout by region or team, and pre-registered pre/post windows all give you something honest to compare against, even though none of them is a perfect laboratory experiment. Each is imperfect; all are better than attributing a good quarter to the launch that happened during it, which is the default and misleading conclusion an organization reaches when no counterfactual was ever set up. Choose the method that fits your regulatory and operational constraints, but choose one deliberately rather than defaulting to a before-and-after comparison with no control.
Confounders are the silent killer of a good baseline
A carefully chosen baseline can still produce a misleading result if a rate change, a reorganization, or a seasonal shift happens during the measurement window and nobody records it. Logging these events as they occur, rather than trying to remember them months later when someone questions the result, is what lets an analyst separate the effect of the AI system from everything else that happened to be moving in the same period. This is inexpensive to do and is the step most likely to be skipped under launch pressure.
In practice
- Name the baseline in writing before development starts
- Hold out a control group where regulation and fairness allow
- Record confounders — reorganizations, rate changes, seasonality — as they happen
- Choose a staged rollout or pre-registered window deliberately, not by default
