Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning

Researchers introduce the Independent Combinatorial Tokens (ICT) framework to improve Large Language Model reasoning by addressing entropy collapse and explosion problems in reinforcement learning. Using Jensen-Shannon divergence to identify critical token branching points, ICT achieves 4.58% average improvement in pass@4 scores across math, commonsense, and Olympiad benchmarks on Qwen models.

The research addresses a fundamental stability problem in training LLMs for complex reasoning tasks. Previous reinforcement learning approaches with verifiable rewards struggled with two opposing failure modes: uniform token updates caused premature convergence to suboptimal solutions through entropy collapse, while excessive entropy maximization led to incoherent reasoning chains. This dichotomy has limited the effectiveness of reasoning-focused LLM training.

The ICT framework represents a technical advancement in how LLM optimization handles uncertainty during training. Rather than applying uniform updates across all tokens, the approach selectively focuses on tokens exhibiting distinctive distributional patterns. By measuring Jensen-Shannon divergence between token logit distributions, the framework identifies these critical branching points that genuinely influence reasoning quality. The dual entropy regulation—reducing Shannon entropy while controlling Rényi entropy—provides theoretical justification for why selective token updates improve training stability.

For the AI development community, these findings suggest that optimization efficiency improvements compound across model scales. The consistent 4.58% average gain across Qwen2.5 variants (0.5B to 7B parameters) and multiple benchmark domains indicates the approach generalizes well rather than overfitting to specific tasks. The 14.9% maximum improvement on particular benchmarks hints at substantial performance gains on harder reasoning problems.

Developers building reasoning-focused LLMs should monitor whether this technique becomes standard practice in reinforcement learning pipelines. The ability to improve reasoning accuracy while maintaining training stability addresses practical deployment concerns. Future work likely explores applying ICT to larger models and whether insights transfer to other LLM architectures beyond Qwen, particularly for specialized reasoning applications in mathematics, coding, and complex problem-solving.

• →ICT framework solves entropy collapse and explosion by selectively updating only top 10% of distinctive tokens, achieving 4.58% average pass@4 improvement
• →Jensen-Shannon divergence identifies critical token branching points that genuinely improve reasoning rather than uniform token optimization
• →Dual entropy regulation prevents over-concentrated token generation while maintaining exploration, improving training stability across model scales
• →Consistent improvements across Qwen2.5 variants and seven benchmarks spanning math, commonsense, and Olympiad problems demonstrate generalization capability
• →Selective token optimization reduces computational overhead while improving reasoning accuracy, with potential applications in production LLM training