Do LLMs Benefit From Their Own Words?

Research reveals that large language models don't significantly benefit from conditioning on their own previous responses in multi-turn conversations. The study found that omitting assistant history can reduce context lengths by up to 10x while maintaining response quality, and in some cases even improves performance by avoiding context pollution where models over-condition on previous responses.

• →Removing prior assistant responses doesn't affect response quality on a large fraction of conversation turns.
• →Multi-turn conversations contain 36.4% self-contained prompts that don't require assistant history.
• →Context pollution occurs when models over-condition on previous responses, introducing errors and hallucinations.
• →Omitting assistant history can reduce cumulative context lengths by up to 10x.
• →Selective context filtering that omits assistant-side context can improve response quality while reducing memory consumption.