Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality

Researchers introduce WikiProfile, a benchmark that reframes LLM factuality failures as either missing knowledge or poor recall of encoded information. Analysis of 13 models shows frontier models encode 95-98% of facts but struggle significantly with recall, suggesting future improvements depend less on scaling and more on better knowledge access mechanisms.

This research addresses a fundamental misunderstanding in how the AI community evaluates language model reliability. Rather than simply measuring whether models produce correct answers, the authors distinguish between two failure modes: facts that were never learned (encoding failures) versus facts the model learned but cannot reliably retrieve (recall failures). This distinction matters because it suggests different solutions—encoding failures require more training data, while recall failures may be solvable through better prompting or reasoning techniques.

The WikiProfile benchmark leverages automated evaluation grounded in web search to profile 4 million responses across major models including GPT-5 and Gemini-3. The finding that frontier models achieve near-saturation on encoding—95-98% of facts are successfully learned—is striking because it indicates we've largely solved the knowledge acquisition problem for these systems. The real bottleneck emerges in recall accessibility, where models fail to produce facts they demonstrably encode.

For the AI development community, these findings redirect optimization efforts. Traditional scaling approaches that primarily add more training data show diminishing returns if encoding is already saturated. Instead, the research validates emerging techniques like chain-of-thought reasoning and inference-time computation ("thinking"), which can recover substantial portions of recall failures. This aligns with recent industry shifts toward longer inference chains and reasoning-focused architectures.

The systematic patterns—recall failures disproportionately affecting long-tail facts and reverse-question formulations—provide concrete targets for future work. Developers building production systems should consider these asymmetries when designing safety evaluations and user-facing interfaces. The research suggests that reliability improvements may increasingly come from architectural innovations rather than raw model scaling.

• →Frontier LLMs encode 95-98% of factual knowledge, making knowledge acquisition nearly saturated
• →Most factuality errors stem from poor recall of encoded facts, not missing knowledge
• →Recall failures are systematic and disproportionately affect long-tail facts and reversed questions
• →Inference-time computation (thinking) substantially improves recall and recovers failed retrievals
• →Future LLM improvements may rely more on access mechanisms than scaling training data