Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models

Researchers introduce Mutual Reinforcement Learning, a framework enabling heterogeneous language models to share training experiences while maintaining separate parameters and tokenizers. The system uses three mechanisms—Shared Experience Exchange, Multi-Worker Resource Allocation, and a Tokenizer Heterogeneity Layer—to coordinate reinforcement learning across incompatible model architectures, with outcome-level success transfer showing the best stability-support trade-off.

This research addresses a fundamental challenge in collaborative AI development: how different language models with incompatible architectures can learn from each other's training experiences. Mutual Reinforcement Learning (MRL) enables concurrent post-training across heterogeneous models by creating a standardized exchange protocol that transcends architectural differences. The framework's innovation lies in its Tokenizer Heterogeneity Layer, which solves the critical problem of aligning token-level information across models using different vocabularies—a barrier that previously prevented meaningful experience sharing.

The research emerges from the broader AI efficiency movement, where collaborative training approaches could reduce computational costs and accelerate model improvement. As organizations deploy increasingly diverse model families for different tasks, frameworks enabling inter-model knowledge transfer become strategically valuable. The three instantiated mechanisms—Peer Rollout Pooling, Cross-Policy GRPO Advantage Sharing, and Success-Gated Transfer—represent different abstraction levels for sharing information, from raw training data to abstract success outcomes.

This work carries implications for AI development infrastructure and multi-agent learning systems. If successfully deployed, MRL could reduce the computational overhead of training multiple models independently, enabling more efficient resource allocation across large AI labs and federated learning scenarios. The framework's flexibility supports different model families collaborating simultaneously, which aligns with industry trends toward heterogeneous deployment architectures.

The contextual-bandit analysis characterizing stability-support trade-offs suggests outcome-level sharing (Success-Gated Transfer) offers the optimal balance, providing direction for future collaborative training implementations. Future work should focus on empirical validation at scale and integration with production training pipelines.

• →Mutual Reinforcement Learning enables heterogeneous language models to share training experiences across incompatible architectures and tokenizers.
• →The Tokenizer Heterogeneity Layer solves vocabulary alignment by retokenizing text and aligning token-level traces across different models.
• →Success-Gated Transfer for outcome-level sharing provides the best trade-off between stability and effective knowledge transfer.
• →The framework reduces computational costs of training multiple model families independently through collaborative experience exchange.
• →Three distinct sharing mechanisms operate at different abstraction levels: data, value, and outcome sharing with measurable stability-support trade-offs.