Probing for Knowledge Attribution in Large Language Models

Researchers developed a method to identify whether large language model outputs come from user prompts or internal training data, addressing the problem of AI hallucinations. Their linear classifier probe achieved up to 96% accuracy in determining knowledge sources, with attribution mismatches increasing error rates by up to 70%.

• →New probe method can predict with 96% accuracy whether LLM outputs derive from user context or internal model weights.
• →AttriWiki dataset enables self-supervised training by having models recall information from memory versus reading from context.
• →Attribution mismatches directly correlate with unfaithful AI responses, increasing error rates by up to 70%.
• →The technique transfers well across different model architectures including Llama, Mistral, and Qwen.
• →Even with correct attribution identification, models may still generate incorrect responses, indicating need for broader detection frameworks.