Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

Researchers propose Bottom-up Policy Optimization (BuPO), a novel reinforcement learning approach that optimizes internal layers of language models rather than treating them as unified policies. The study reveals that LLMs contain distinct internal policy structures with different entropy patterns across layers, offering new insights into how transformer-based models process reasoning tasks.

This research addresses a fundamental gap in how reinforcement learning is applied to large language models. Traditional RL approaches treat LLMs as black-box unified policies, but this study demonstrates that models contain sophisticated internal mechanisms that evolve predictably across layers. The entropy analysis reveals that early layers engage in high-entropy exploration while top layers perform deterministic refinement, suggesting a hierarchical reasoning structure within single models.

The findings highlight important differences between model architectures. Qwen exhibits explicit progressive reasoning across layers, while Llama shows abrupt convergence patterns. This architectural insight is significant because it suggests that optimization strategies should be tailored to each model's internal structure rather than applying uniform training approaches.

BuPO's core innovation lies in optimizing internal layers during early training stages, forcing lower layers to capture high-level reasoning representations before final layers process them. This bottom-up reconstruction of reasoning foundations challenges conventional wisdom that optimization should target output layers. The approach demonstrates effectiveness on complex reasoning benchmarks, indicating practical value beyond theoretical insights.

For the AI development community, this research opens new optimization pathways for making language models more efficient and capable at reasoning tasks. Rather than scaling model size or training data, developers might improve performance by understanding and leveraging internal policy structures. This has implications for both training efficiency and model interpretability, two areas where current approaches struggle to scale sustainably.

• →LLMs contain decomposable internal layer policies with distinct entropy patterns across layers that follow predictable evolution patterns.
• →Different model architectures (Qwen vs Llama) exhibit fundamentally different internal policy structures, suggesting optimization should be architecture-specific.
• →Bottom-up Policy Optimization improves complex reasoning by optimizing early layers to capture high-level representations before top layers refine outputs.
• →Internal policy analysis reveals that early layers perform exploration while final layers perform deterministic refinement, mirroring exploration-exploitation principles.
• →The research suggests improvements to language model reasoning can be achieved through internal mechanism optimization rather than just scaling.