Operations
Hallucination
Also called: confabulation, fabrication
Definition
A hallucination is a fluent, confident output that is not supported by any source — not a malfunction, but the predictable behaviour of a system built to produce plausible text.
Treating hallucination as a bug to be patched leads to endless prompt tinkering that never fully resolves the problem, because the behavior is intrinsic to how the model works rather than an isolated defect. Treating it as a property of the technology leads to architecture: ground the model in retrieved evidence, require citations, and route anything unsupported to a person. The goal is not zero hallucination but zero unreviewed hallucination reaching a decision that matters.
Where it actually hurts
Drafting an email that a human reads before sending is a low-stakes error — the reader catches an obviously wrong fact before anything happens. A fabricated policy limit inside an automated coverage decision is a loss event, because there is no human between the fabrication and the money moving. The same failure rate carries wildly different cost depending on where in the workflow it lands, which is why the correct question is never 'how often does it hallucinate' alone, but 'what happens when it does, at this specific point in this specific workflow.'
Grounding beats scolding
Instructions like 'do not make things up' change little, because the model has no internal mechanism to check its own output against a store of verified facts; the instruction is just more text to predict around. Supplying the source text, constraining the answer to it, and rejecting responses without a citation change a great deal, because these are architectural constraints rather than requests for better behavior, and they work even when the model itself has no idea whether a given claim is true.
Measuring it separately from quality
A response can be well-written, well-organized, and entirely ungrounded, which is why groundedness needs its own score in an eval rather than being folded into a general 'quality' rating that a fluent hallucination will pass easily. Building a specific check — does every material claim trace back to a retrieved passage — catches failures that a subjective quality review, which tends to reward confident and readable prose, will consistently miss. This distinction is one of the most common gaps in eval design for language-heavy workflows.
In practice
- Require every material claim in an output to cite retrieved source text
- Score groundedness separately from usefulness in your evals
- Put a human in front of any hallucination that would be expensive
- Build a dedicated groundedness check rather than folding it into a general quality score
