It Read Tens of Thousands of Words, So Why Did It Still Miss Key Information: Lost in the Middle

You’ve likely encountered this scenario: You feed a 30-page requirements document into a large language model (LLM), asking it to summarize the content and list acceptance criteria. It returns a beautifully structured, full-page response, replete with every technical term from its training corpus—yet it conveniently omits that one sentence on page 14: “Refunds must be initiated within 48 hours of order creation.”

What’s even more frustrating is that larger models aren’t necessarily more reliable. A 2023 Stanford paper, *Lost in the Middle*, tested a batch of then-mainstream open-source models by placing key information at different positions within long documents to measure retrieval accuracy. The result was a familiar U-shaped curve: When information appeared at the beginning or end, the probability of a correct answer was significantly higher; when placed in the middle, accuracy dropped sharply, with some models seeing their performance cut in half. Subsequent long-context models have flattened much of this U-curve, but they haven’t eliminated it entirely.

The reasons for this can be explained directly through the mechanics of inference engines, rather than attributed to mysticism.

First, attention distribution is uneven. When a Transformer generates the $n$-th token, the KV cache holds all previous positions, each participating in softmax normalization. As the context lengthens, the total attention budget is spread thinner. Coupled with positional encodings that are inherently less favorable to tokens far from the generation point, the actual attention weight assigned to middle chunks tends to be low. The information is still in the KV cache, but it isn’t “loud” enough.

Second, there is a mismatch between training distribution and real-world usage. The paper includes a clean controlled experiment: When asked to directly recall the $k$-th fact from memory (closed-book), models showed only a slight drop in accuracy. However, once switched to an open-book setting—where the fact was embedded within a longer article before being queried—accuracy declined significantly. In other words, the issue isn’t that the model “can’t remember the 40th fact,” but rather that it “knows the 40th fact but doesn’t know where to look amidst a heap of irrelevant context.” During long-text training, effective information is predominantly clustered at the beginning and end of documents, while the middle sections often resemble noise in pre-training corpora. Consequently, the model learns the cost-effectiveness of “looking at the ends” rather than performing uniform retrieval.

This phenomenon has three direct implications for production systems.

1. **Don’t bury key requirements in the middle.** Hard constraints in long system prompts (such as output format, red lines, or mandatory data references) should be placed at the very beginning or the very end, leaving the middle for background material. This is a zero-cost change that directly addresses the underlying mechanism.

2. **Split instead of stuffing.** Documents can be chunked and subjected to vector retrieval, injecting only the top-$k$ relevant paragraphs (along with page numbers) into the context. While retrieval isn’t perfect, the relevant segments usually amount to no more than a few thousand tokens. In this scenario, the model effectively faces a “short document,” and the performance penalty from *Lost in the Middle* largely disappears. Engineering-wise, this is called RAG (Retrieval-Augmented Generation), but its essence isn’t just “saving tokens”; it’s about reshaping the positional distribution of information.

3. **Layer your prompts and materials.** Task instructions, constraints, and examples constitute one category of information that should stay close to the generation point; factual material constitutes another. Regardless of length, the former should always be placed at the head or tail, while the latter should be left to retrieval mechanisms. Many bugs labeled as “the model isn’t following instructions” turn out not to be capability issues, but rather cases where the instruction was buried as the 38th paragraph among 50 blocks of data. To check if you’re falling victim to this curve, use a rudimentary method: Extract 10 hard requirements from your long document, manually label their quantile positions in the original text (first quartile, middle half, last quartile), and ask the model to recite them one by one. Bucket the miss rates by position. If the miss rate in the middle bucket is significantly higher than at the ends, it means the structure of your input material conflicts with the model’s positional bias. Rearranging and rerunning will confirm this.

Window size determines “how much it can fit,” while retrieval and layout determine “whether it remains usable once fitted.” The former is a hardware metric; the latter is an engineering problem. And in production environments, most failures occur due to the latter.

The next post will discuss why attention distribution is biased toward the tail, and why sliding window attention causes the state of long conversations to drift subtly.