Operations
Context window
Definition
The context window is the amount of text a model can consider at once — the working memory that bounds what any single call can take into account.
Large windows tempt teams to paste in everything, but relevance degrades as volume grows and cost scales with it. A tightly retrieved set of passages usually outperforms an entire document dump, and it is far cheaper. Window size is a constraint to design around, not a feature to max out, and treating it as a substitute for good retrieval design is one of the more expensive habits a team can develop.
More context is not more accuracy
Models attend unevenly across long inputs, and important detail buried in the middle of a very long prompt is routinely missed even when it is technically within the window. Curation beats volume, because a model given ten carefully chosen relevant passages will typically outperform the same model given the same ten passages buried inside two hundred pages of loosely related material, since the signal-to-noise ratio of the prompt matters more than its raw size.
Windows are a cost lever
Every token in the window is billed on every call, whether or not that token ends up mattering to the answer. A workflow that quietly grew its prompt by ten pages over several months of well-intentioned additions has quietly multiplied its unit cost by a similar factor, often without anyone noticing until a finance review asks why the inference bill has grown faster than usage.
Long context does not remove the need for retrieval
As context windows have grown, some teams have concluded that retrieval is no longer necessary and everything can simply be pasted into the prompt. This trades a retrieval problem for a cost and reliability problem, since dumping an entire document set into every call is both expensive and, as noted above, no more accurate than a well-curated retrieval step. A larger window is useful headroom for genuinely complex tasks, not a license to skip the retrieval discipline that keeps a workflow both accurate and affordable.
In practice
- Retrieve the smallest sufficient evidence rather than the whole file
- Track average prompt size per workflow as a cost metric
- Re-test accuracy whenever the prompt structure changes materially
- Keep the retrieval discipline even as context windows grow larger
