Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison

Researchers introduce a benchmark for evaluating how AI systems handle conflicting information across multiple memory sources, addressing a critical gap in testing personal AI agents. The study compares various approaches including fusion methods and LLMs, revealing that trained fusion models outperform prompt-based LLMs by 10+ percentage points on accuracy, with selective abstention improving performance further.

This research tackles a fundamental challenge in deploying persistent AI agents: how systems should reason when faced with contradictory or incomplete evidence from multiple sources. Unlike traditional QA benchmarks that assume clean, single-source data, this work reflects real-world conditions where personal AI systems encounter conflicting information across emails, calendars, chats, and other data streams. The gap between training data assumptions and deployment reality has long plagued AI evaluation, making this testbed a meaningful contribution to the field.

The benchmark's sophistication lies in its controlled design—18 question templates, 8 reasoning types, and deterministic ground truth enable precise diagnostics of where systems fail. By separating retrieval errors from conflict-resolution errors, researchers can pinpoint whether failures stem from accessing wrong information or from poor decision-making when multiple sources disagree. This granularity is absent from most existing benchmarks, which conflate these failure modes.

The results reveal interesting trade-offs: trained fusion methods achieve higher accuracy (80.3%) but lower coverage, while LLMs maintain broader coverage (95.4%) at lower accuracy (71.0%). This suggests different architectural approaches suit different use cases—critical systems might prioritize accuracy with selective abstention, while exploratory applications benefit from LLM flexibility. The 10+ percentage point gap between fusion and prompt-only LLMs challenges the assumption that frontier LLMs solve all reasoning problems through scale alone.

The release of code, data, and model outputs enables community iteration on conflict-resolution methods, likely attracting attention from researchers building knowledge-base systems, retrieval-augmented generation pipelines, and personal knowledge management tools. Future work should examine performance on naturally occurring conflicts rather than synthetic distortions.

• →Trained fusion methods outperform frontier LLMs by 10.3 percentage points on conflicting multi-source reasoning tasks.
• →The benchmark separates retrieval failures from conflict-resolution failures, enabling precise diagnostic analysis of system weaknesses.
• →Selective abstention allows systems to trade recall for precision, with the best resolver reaching 85.3% accuracy at 78.3% coverage.
• →Different model architectures show varying strengths across reasoning types, suggesting no single approach optimizes all scenarios.
• →Open release of benchmark, code, and cached outputs accelerates research into personal AI memory systems.