Revisiting Mixture Policies in Entropy-Regularized Actor-Critic

Researchers propose a marginalized reparameterization (MRP) estimator to enable practical use of mixture policies in reinforcement learning, addressing a long-standing gap between theoretical potential and practical implementation. By reducing variance compared to likelihood-ratio methods, MRP mixture policies achieve performance parity with standard Gaussian policies while offering greater flexibility in continuous action spaces.

This research tackles a fundamental inefficiency in modern reinforcement learning systems. Mixture policies—which combine multiple probability distributions—theoretically provide superior representational capacity compared to single Gaussian policies, yet remain absent from production algorithms like Soft Actor-Critic (SAC). The disconnect stems from a technical limitation: unlike Gaussian policies, mixtures lack an efficient low-variance gradient estimation method, forcing practitioners to rely on high-variance likelihood-ratio approaches that negate theoretical advantages.

The proposed marginalized reparameterization estimator solves this bottleneck by enabling low-variance gradient computation for mixture policies. The researchers provide formal variance analysis proving MRP outperforms standard likelihood-ratio approaches, then validate claims across diverse benchmarks including MuJoCo, DeepMind Control Suite, and MetaWorld. Results demonstrate that mixture policies trained with MRP match or occasionally exceed Gaussian policy performance while maintaining theoretical flexibility benefits.

For the reinforcement learning community, this work bridges a persistent gap between theory and practice. The significance extends beyond academic curiosity—mixture policies enable learning multi-modal behaviors crucial for robotic control, autonomous navigation, and complex decision-making under uncertainty. The empirical finding that MRP mixtures sometimes outperform Gaussian baselines suggests untapped potential in RL algorithm design.

The broader implication concerns algorithm development methodology. The paper demonstrates how identifying and solving specific technical obstacles can unlock dormant theoretical advantages. Future RL research may benefit from systematic examination of other high-potential techniques abandoned due to implementation challenges rather than fundamental limitations.

• →Marginalized reparameterization provides low-variance gradient estimation for mixture policies, addressing the primary barrier to practical adoption.
• →MRP mixture policies achieve performance parity with standard Gaussian policies while offering greater representational flexibility.
• →Theoretical analysis proves mixture policies enhance solution quality and entropy robustness when properly implemented.
• →Empirical results identify specific scenarios where mixture policies demonstrate clear advantages over unimodal alternatives.
• →Technical obstacle removal can elevate theoretically sound but practically abandoned techniques into valuable practical tools.