Beyond Chunking: Discourse-Aware Hierarchical Retrieval for Long Document Question Answering

Researchers present a discourse-aware hierarchical framework that uses rhetorical structure theory (RST) to improve long-document question answering systems. Rather than treating documents as flat sequences, the approach leverages natural discourse structures to enhance retrieval accuracy across multiple languages and document types.

This research addresses a fundamental limitation in current long-document question answering systems, which typically rely on naive chunking strategies that ignore how documents are naturally organized. By incorporating rhetorical structure theory, the framework recognizes that human comprehension follows discourse patterns—transitions between ideas, hierarchical relationships between concepts, and logical connections between sections. The innovation combines three technical components: language-universal discourse parsing that works across linguistic boundaries, LLM-enhanced representations of discourse nodes that capture both structural and semantic information, and hierarchical retrieval mechanisms that prioritize relevant structural paths through documents.

The work represents incremental but meaningful progress in natural language understanding. Discourse-aware approaches have been theoretically sound but computationally challenging to implement at scale. This research demonstrates that integrating structural linguistics with modern language models yields consistent improvements across diverse datasets and languages. The framework's robustness across document types suggests the approach generalizes beyond narrow use cases, addressing real-world heterogeneity in document structure and language.

For the AI industry, this development signals growing sophistication in retrieval-augmented generation (RAG) systems, which underpin many enterprise AI applications requiring access to proprietary documents. Organizations deploying question answering systems over technical documentation, legal contracts, or research papers could benefit from improved accuracy. The multilingual capability particularly matters for global enterprises managing polyglot document repositories. However, the research remains academic; practical deployment requires integration with existing RAG pipelines and benchmarking against production systems. The work doesn't solve fundamental challenges around computational efficiency or real-time performance at scale.

• →Discourse-aware hierarchical retrieval improves long-document QA by leveraging natural document structure rather than flat chunking approaches.
• →The framework combines rhetorical structure theory with LLM-enhanced representations to bridge linguistic structure and semantic meaning.
• →Consistent improvements demonstrated across four datasets and multiple languages, suggesting strong generalization capabilities.
• →Particularly relevant for enterprise RAG systems processing technical documentation, legal contracts, and research materials.
• →Addresses a real gap in current retrieval systems but requires further optimization for production-scale deployment.