Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries

Researchers evaluated whether large language models understand long-form narratives similarly to humans by comparing summaries of 150 novels written by humans and nine state-of-the-art LLMs. The study found that LLMs focus disproportionately on story endings rather than distributing attention like human readers, revealing gaps in narrative comprehension despite expanded context windows.

This research addresses a fundamental limitation in modern language models: the gap between theoretical capacity and practical comprehension. While context windows have expanded dramatically, the ability to meaningfully integrate information across long documents remains uneven. By using novel summaries as a benchmark, researchers created an elegant methodology to measure how well models understand narrative structure—a task requiring genuine conceptual engagement rather than surface-level pattern matching.

The findings reveal a specific failure mode: LLMs exhibit a recency bias, emphasizing story conclusions at the expense of balanced narrative comprehension. This contrasts sharply with human summarization, where writers identify thematically important moments distributed throughout texts. This disparity likely stems from how transformer architectures process sequential information and allocate attention across tokens, suggesting the problem is architectural rather than simply a matter of training data or model scale.

For the AI industry, this research has direct implications for deployment scenarios requiring genuine document comprehension—legal analysis, research synthesis, and long-form content generation. The insight that increased context length doesn't automatically improve comprehension challenges assumptions underlying recent scaling efforts. The released dataset provides a valuable benchmark for evaluating future models and measuring progress on this specific weakness.

Developers building production systems relying on long-document understanding should account for these biases. Future LLM improvements might require architectural modifications beyond scaling, including mechanisms that better distribute attention across narrative structures rather than clustering at endpoints.

• →LLMs distribute attention toward story endings rather than mirroring human patterns of identifying narratively important moments
• →Extended context windows have not proportionally improved long-form narrative comprehension despite theoretical capacity expansion
• →Novel summaries serve as an effective benchmark for measuring genuine conceptual engagement versus surface-level processing
• →The comprehension gap likely stems from transformer architecture limitations rather than insufficient training data
• →A new dataset of 150 human-novel summaries aligned with chapters provides a resource for evaluating and improving narrative understanding