Escaping the Verifier: Learning to Reason via Demonstrations

Researchers introduce RARO, a new training method that enables Large Language Models to develop strong reasoning capabilities using only expert demonstrations, without requiring task-specific verifiers. The approach uses adversarial learning between a policy and critic to achieve significant performance improvements across multiple reasoning tasks.

RARO addresses a fundamental challenge in AI training: most reasoning tasks lack reliable verifiers, yet possess abundant expert demonstrations that current methods underutilize. This research bridges that gap through inverse reinforcement learning, where a critic learns to distinguish expert from policy-generated solutions while the policy simultaneously learns to mimic expert reasoning. The adversarial dynamic creates a self-improving system without external validation signals.

Traditionally, reasoning-focused AI training relies heavily on reinforcement learning with task-specific verifiers—essentially automated graders that evaluate whether a solution is correct. This approach works well for tasks with clear right answers, but many real-world reasoning problems resist simple verification. Poetry writing, mathematical theorem proving, and complex planning tasks often lack objective correctness criteria. RARO's framework leverages the insight that expert demonstrations contain implicit quality signals that can be extracted through adversarial training, sidestepping the verifier bottleneck entirely.

The empirical results demonstrate substantial improvements: 13.7% on Countdown tasks, 8.2% on DeepMath problems, and 19.1% on poetry evaluation against expert baselines. These gains suggest the method effectively captures reasoning patterns from demonstrations. The scalability improvements comparable to verifier-based approaches indicate RARO maintains performance as model size increases, addressing a critical concern for deployment.

This development broadens the landscape of viable reasoning training methodologies. Organizations working with reasoning-intensive tasks lacking automated evaluation can now consider demonstration-based approaches. The technique's robustness across disparate domains—from mathematical computation to creative writing—suggests wider applicability across AI systems requiring complex inference without clear verification mechanisms.

• →RARO enables reasoning training using only expert demonstrations through adversarial learning, eliminating the need for task-specific verifiers.
• →The method achieves significant performance gains: +13.7% accuracy on Countdown, +8.2% on DeepMath, and +19.1% win-rate on Poetry Writing tasks.
• →Adversarial training between policy and critic creates a self-improving system that extracts implicit quality signals from expert examples.
• →RARO demonstrates robust scaling comparable to verifier-based reinforcement learning approaches across different model sizes.
• →The framework addresses real-world limitations where many reasoning-intensive tasks lack reliable automated verifiers despite abundant expert demonstrations.