An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs

Researchers propose ITEM, an iterative utility judgment framework that enhances retrieval-augmented generation (RAG) systems by aligning with philosophical principles of relevance. The framework improves how large language models prioritize and process information from retrieval results, demonstrating measurable improvements across multiple benchmarks in ranking, utility assessment, and answer generation.

This research addresses a fundamental challenge in retrieval-augmented generation systems: distinguishing between information that is relevant to a query and information that is actually useful for generating accurate answers. While traditional IR systems focus on relevance ranking, RAG systems must balance computational constraints with answer quality, making utility judgment increasingly critical as LLM input bandwidth remains limited. The connection to Schutzian philosophy provides theoretical grounding for a problem previously treated primarily as an engineering challenge.

The ITEM framework's iterative approach reflects broader trends in AI research toward more sophisticated context handling. As LLMs become more capable but expensive to run at scale, efficient information filtering becomes economically important. The framework's alignment of three cognitive levels—retrieval ranking, utility judgment, and answer generation—mirrors how advanced AI systems are increasingly structured around human cognition patterns rather than purely statistical optimization.

For the AI industry, this work has practical implications. Organizations deploying RAG systems could improve answer quality while reducing computational costs by implementing better utility filtering. The experimental validation across multiple datasets (TREC DL, WebAP, GTI-NQ, NQ) suggests the approach generalizes beyond specific domains. This matters particularly for applications where hallucination reduction and factual accuracy are critical, such as healthcare, legal, and financial services.

Future development likely focuses on integrating such utility frameworks into production RAG pipelines and exploring whether the philosophical foundations suggest additional optimization opportunities beyond current implementations.

• →ITEM framework distinguishes between relevance and utility in RAG systems, improving how useful information is prioritized for LLMs.
• →The approach connects philosophical theory of relevance to practical AI engineering, providing theoretical grounding for information filtering.
• →Experimental results show improvements in ranking quality, utility judgment accuracy, and final answer generation across multiple benchmarks.
• →Efficient utility judgment can reduce computational costs while maintaining or improving answer quality in resource-constrained RAG deployments.
• →The framework's success suggests that modeling human cognitive levels improves AI system architecture more effectively than pure statistical optimization.