Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models

Researchers introduce chain-of-illocution (CoI) prompting to improve source faithfulness in retrieval-augmented language models, achieving up to 63% gains in source adherence for programming education tasks. The study reveals that standard RAG systems exhibit low fidelity to source materials, with non-RAG models performing worse, while a user study confirms improved faithfulness does not compromise user satisfaction.

This research addresses a critical gap in explainable AI: ensuring that LLM-generated explanations remain grounded in verifiable sources rather than producing plausible-sounding but unsupported claims. The study benchmarks six LLMs against Stack Overflow questions using programming textbooks as authoritative sources, revealing a stark problem: non-RAG models show 0% median source adherence, while baseline RAG systems achieve only 22-40%. This matters because users often cannot distinguish between evidence-backed and hallucinated explanations, creating risks in high-stakes domains like education.

The proposed solution draws from linguistic theory—specifically Achinstein's illocutionary theory—to frame explanations as communicative acts with distinct intentions. Chain-of-illocution prompting expands queries into implicit explanatory questions before retrieval, effectively guiding the model to surface relevant source material. The approach demonstrates statistically significant improvements, with some models showing 63% gains in source adherence.

For the AI industry, this represents progress toward more trustworthy systems in knowledge-intensive applications. The user study's 165 retained participants showed that improved source fidelity did not degrade perceived quality, suggesting no inherent trade-off between faithfulness and user experience. However, the moderate absolute adherence rates (even post-improvement) and inconsistent gains across models indicate the problem remains partially unsolved.

Future work should explore whether these techniques scale to other domains beyond programming education and whether they generalize to smaller, more resource-constrained models. The theoretical grounding in illocutionary theory also opens opportunities for developing more principled prompting strategies.

• →Chain-of-illocution prompting achieves up to 63% improvements in source adherence for RAG systems in educational contexts
• →Standard RAG baselines show surprisingly low source faithfulness (22-40% median), indicating retrieval alone is insufficient for grounding explanations
• →Improving source fidelity does not harm user satisfaction, relevance perception, or perceived correctness according to 165-participant user study
• →Theoretical grounding in illocutionary theory provides a linguistic framework for designing more faithful explanation generation
• →Absolute adherence remains moderate across models, suggesting source faithfulness remains a partially solved problem in LLMs