Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning

Researchers introduce Generative Adversarial Reasoner, a new training framework that improves LLM mathematical reasoning by using adversarial reinforcement learning between a reasoner and discriminator model. The method achieved significant performance gains on mathematical benchmarks, improving DeepSeek models by 7-10 percentage points on AIME24 tests.

• →Generative Adversarial Reasoner uses adversarial reinforcement learning to co-train an LLM reasoner with an LLM-based discriminator.
• →The framework addresses common LLM reasoning errors like incorrect calculations and invalid logical steps through dense step-level rewards.
• →Testing showed 7.3 point improvement for DeepSeek-R1-Distill-Qwen-7B and 10.0 point improvement for DeepSeek-R1-Distill-Llama-8B on AIME24.
• →The method provides better credit assignment and sample efficiency compared to standard reinforcement learning approaches.
• →The modular discriminator enables flexible reward shaping for various objectives including teacher distillation and proof-based reasoning.