OISD: On-Policy Internal Self-Distillation of Language Models

Researchers introduce OISD, a new reinforcement learning framework that improves language model reasoning by having the final layer act as an internal teacher to guide intermediate layers through logit and attention alignment. The method demonstrates consistent improvements across mathematical reasoning tasks without requiring external data.

The OISD framework addresses a fundamental limitation in current RL post-training approaches for language models. While existing methods focus on optimizing final outputs using sparse rewards, they fail to leverage the rich predictive signals embedded throughout intermediate network layers. This research proposes using the final layer as a detached internal teacher during training, transferring knowledge backward through the network via two complementary mechanisms: logit alignment captures high-level reasoning patterns, while attention alignment enforces consistent focus patterns across layers.

This approach builds on broader trends in machine learning toward more efficient training paradigms. Self-distillation techniques have proven valuable in other domains, but applying them to intermediate layers during RL optimization represents a novel contribution. The use of signed advantage-weighted Jensen-Shannon alignment ensures the framework maintains policy consistency while distilling informative representations, avoiding the common pitfall of distillation degrading the original policy's performance.

The significance extends beyond academic interest. Language models with stronger reasoning capabilities directly impact commercial applications in code generation, mathematical problem-solving, and complex instruction following. Improvements in reasoning efficiency could reduce computational requirements during fine-tuning, making advanced model customization more accessible to smaller organizations. The framework's compatibility with Group Relative Policy Optimization suggests practical implementation pathways for existing training infrastructures.

Future developments will likely explore whether OISD principles apply to other domains beyond mathematical reasoning and whether the internal distillation mechanism scales to larger models. The promised code release should enable rapid validation and adoption across research and industrial settings, potentially establishing new standards for reasoning-focused model training.

• →OISD uses the final layer as an internal teacher to guide intermediate layers through logit and attention alignment mechanisms.
• →The framework improves mathematical reasoning performance without requiring external privileged information or additional training data.
• →Signed advantage-weighted Jensen-Shannon alignment preserves policy consistency while distilling intermediate representations.
• →The approach addresses the overlooked potential of predictive signals in intermediate layers during RL post-training.
• →Successful validation across multiple mathematical reasoning tasks suggests practical applicability to production language model training.