Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty

Researchers developed an information-theoretic framework to explain 'Aha moments' in large language models during reasoning tasks. The study reveals that strong reasoning performance stems from uncertainty externalization rather than specific tokens, decomposing LLM reasoning into procedural information and epistemic verbalization.

• →LLMs exhibit 'Aha moments' during reasoning through apparent self-correction following uncertainty tokens like 'Wait'.
• →The research introduces a framework decomposing reasoning into procedural information and epistemic verbalization.
• →Purely procedural reasoning can become informationally stagnant without uncertainty externalization.
• →Strong reasoning performance is driven by uncertainty externalization rather than specific surface tokens.
• →The framework provides insights for future reasoning model design and unifies prior research findings.