Governance
Explainability
Also called: interpretability, reason codes
Definition
Explainability is the ability to state, in terms a regulator or a customer accepts, why a specific decision came out the way it did.
In regulated decisions the explanation is part of the product. A declined application, a denied claim, or an adverse pricing action needs a defensible reason grounded in permitted factors — not a narrative generated after the fact to sound plausible. Systems that cannot produce that record are not deployable in those workflows regardless of accuracy, because accuracy alone does not satisfy a regulator asking for the specific, defensible basis of an individual decision.
Post-hoc narrative is not a reason code
Asking a model to explain its own output produces a plausible story that may bear no relationship to the actual computation that produced the decision, since the explanation and the decision were not generated by the same process and the model has no genuine introspective access to its own reasoning. Explanations must be derived from the decision inputs, not authored alongside the answer, because a fabricated-but-plausible explanation is arguably worse than no explanation at all when it is later scrutinized.
Build the record at decision time
Capture the inputs, the retrieved evidence, the model version, and the applied rules when the decision is made, while all of that context is still available and cheap to record. Reconstructing it eighteen months later during an examination is not realistic, because the underlying model may have changed, the retrieved documents may have been updated or deleted, and nobody will remember the specific reasoning that applied to one file out of thousands processed that quarter.
Explainability and accuracy are not the same test
A highly accurate model that cannot produce a defensible individual explanation is not automatically deployable in a regulated decision, and a less accurate but fully explainable system is sometimes the correct choice for a specific workflow precisely because of that constraint. Teams evaluating a new system should test explainability as its own criterion, separate from an accuracy score, because a system can pass one test and fail the other, and both failures carry real consequence in a regulated context.
In practice
- Persist inputs, evidence, model version, and thresholds per decision
- Restrict permitted factors explicitly and test for proxies
- Rehearse an examination request against your actual stored records
- Score explainability as its own criterion, separate from accuracy, during evaluation
