Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light

Researchers challenge three foundational assumptions in reinforcement learning—treating environments as Markov processes, learning as policy optimization, and agents as scalar reward maximizers—proposing instead a framework grounded in evolutionary dynamics and thermodynamic theories of agency. The work suggests reconceptualizing agent learning as adaptation rather than optimization, with goals extending beyond simple reward signals.

This arXiv paper addresses a fundamental theoretical gap in reinforcement learning by questioning assumptions that have guided the field for decades. The authors argue that RL's three core tenets, while computationally useful, lack biological fidelity and formal grounding in what actually constitutes an adaptive agent. This critique matters because as AI systems become more autonomous, understanding agency at a theoretical level becomes increasingly urgent for both safety and capability design.

The research draws on artificial life and evolutionary computation to propose alternatives. Rather than treating agents as reward-maximizing optimizers, the framework positions them as adaptive systems operating under constraints derived from thermodynamic principles of origin-of-life research. Open-ended novelty search replaces scalar reward optimization as a model for goal-directed behavior, enabling systems to explore solution spaces without predetermined objective functions.

For AI practitioners, this work suggests potential improvements in agent robustness and generalization. Current RL systems often fail when deployed beyond their training distributions; treating adaptation as an evolutionary process informed by thermodynamic principles could yield more resilient architectures. The implicit critique of reward-based learning also has safety implications—agents optimizing narrow reward signals can exhibit unintended behaviors, whereas adaptation-based frameworks might naturally encourage more balanced exploration.

Looking ahead, researchers should watch whether these theoretical insights translate into practical improvements in benchmark performance and real-world deployment. Integration of evolutionary dynamics with modern deep RL architectures remains an open engineering challenge. If successful, this could reshape how AI systems are designed, particularly for long-horizon autonomous tasks where traditional reward shaping proves insufficient.

• →Reinforcement learning's foundational assumptions lack biological grounding and formal theory of agency.
• →Reconceptualizing learning as adaptation rather than optimization could improve agent robustness in novel environments.
• →Open-ended novelty search offers an alternative to scalar reward maximization for goal-directed behavior.
• →Thermodynamic principles from origin-of-life research provide formal foundations for understanding adaptive systems.
• →This theoretical framework could enhance AI safety by moving beyond narrow reward optimization paradigms.