Talking AI — The context window: 99.3% accuracy on one page, 69.7% on a long file

A context window is how much text an AI can read at once. A vendor advertising “128,000 tokens” is describing a capacity, not a guarantee of quality across that whole length. A study presented at ICML 2025 tested thirteen models that all claim this large capacity on progressively longer documents: by 32,000 tokens of context (already a long report), eleven of them fall below half of their initial accuracy, and the best drops from 99.3% correct answers on a short excerpt to 69.7% on a long text. The more pages you ask it to read at once, the more the model is likely to miss what sits in the middle.

So the question to ask a vendor is not “how big is your window?” but “how well does accuracy hold up on my longest files, not just on a demo excerpt?” Before handing an AI a long file in one piece, have it tested on your largest real documents and require it to split them into targeted sections rather than process them in a single block.

Source: Modarressi et al., “NoLiMa: Long-Context Evaluation Beyond Literal Matching,” ICML 2025 (13 models tested).