Failure Modes of Large Language Models on Research-Level Mathematics: A Taxonomy and an Empirical Characterisation

Researchers identify four specific failure modes in large language models attempting research-level mathematics: citation fabrication, premise smuggling, silent problem reformulation, and local-to-global compatibility gaps. Testing reveals that premise smuggling—where models assert unjustified claims as fundamental results—persists even when citations are accurate, suggesting retrieval-augmented generation alone cannot solve LLM reasoning failures.

The study exposes a critical vulnerability in how state-of-the-art language models approach complex mathematical reasoning. Rather than merely failing silently, models like Gemini 2.5 Flash generate confident, fluent-sounding proofs containing unsubstantiated leaps of logic. The research moves beyond surface-level criticism by mapping failure modes with surgical precision, revealing that the most dangerous failure—premise smuggling—operates orthogonally to solutions the industry has prioritized.

This work builds on the "First Proof" benchmark's finding that even the strongest publicly available models consistently fail on research-level mathematics. The taxonomy identifies that models don't just lack knowledge; they mask gaps by asserting claims without justification, presenting partial arguments as established results. This pattern fundamentally differs from hallucinated citations, which fact-checking systems can theoretically catch. Premise smuggling, by contrast, bypasses citation verification entirely because it operates at the logical architecture level.

The implications cut across AI development strategy. Companies betting on retrieval-augmented generation as a fix for model reasoning errors face an uncomfortable reality: their approach addresses only one symptom while leaving the core pathology intact. The finding that 100% of tested proofs contained load-bearing unjustified premises, despite zero confirmed fabricated citations, suggests the problem runs deeper than information access.

The research's most important contribution is its recommendation to shift from detection to prevention through inference-time pipelines. This suggests the path forward requires architectural changes to how models generate mathematical proofs—implementing verification checks during reasoning rather than post-hoc auditing. The work establishes a foundation for building mathematically rigorous AI systems by first understanding precisely where and how current systems fail.

• →Large language models fail at research mathematics by asserting unjustified premises as fundamental results rather than fabricating citations.
• →Premise smuggling—the core failure mode—invisibly bypasses citation verification systems because it operates at the logical layer.
• →All eight test proofs contained at least one load-bearing claim with no justification, yet zero contained confirmed fabricated citations.
• →Retrieval-augmented generation cannot fix premise smuggling, requiring instead inference-time architectural changes to prevent failures.
• →The research shift from failure detection to prevention-focused pipeline design indicates the next evolution in AI reasoning systems.