SocraticPO: Policy Optimization via Interactive Guidance

SocraticPO is a new reinforcement learning framework that improves large language model training by combining natural-language teacher guidance with reward decay, rather than relying solely on scalar outcome rewards. The method shows improvements on scientific reasoning benchmarks while preventing models from exploiting teacher assistance as a shortcut to rewards.

SocraticPO addresses a fundamental challenge in reinforcement learning for language models: scalar rewards like binary correctness signals lack explanatory power. Without understanding *why* an answer is wrong, models develop brittle policies and rely on shortcuts rather than genuine reasoning improvements. This framework introduces Socratic-style guidance where a teacher diagnoses errors and provides targeted corrections, mimicking human tutoring approaches that have proven effective in education.

The innovation lies in pairing guidance with reward decay—correct answers achieved after teacher intervention receive diminished rewards. This prevents models from gaming the system by depending on assistance. The approach maintains compatibility with existing policy-gradient methods like Reinforce++, enabling broad adoption without architectural changes. By accepting only text-level guidance, SocraticPO avoids the logit-access requirements that constrain other approaches, allowing integration of stronger black-box teacher models.

Results on SciKnowEval benchmarks demonstrate tangible improvements over strong RL and self-distillation baselines, with ablations confirming both guidance and reward decay are essential. This matters for developing more reliable AI systems capable of complex reasoning tasks. The framework could accelerate progress in scientific problem-solving and educational applications where reasoning transparency matters.

Looking ahead, the field will monitor whether this approach scales beyond undergraduate-level problems to professional-grade reasoning tasks. Extensions might explore adaptive guidance strategies, dynamic reward decay schedules, and applications in domain-specific reasoning where expert teacher models exist. Success here could reshape how researchers approach policy optimization for knowledge-intensive domains.

• →SocraticPO combines natural-language teacher guidance with reward decay to improve LLM reasoning without encouraging shortcut learning
• →The framework integrates seamlessly with existing policy-gradient methods, enabling broad adoption without architectural redesign
• →Reward decay prevents models from exploiting teacher assistance as a path to easy rewards, forcing genuine reasoning improvement
• →Results exceed strong RL and self-distillation baselines on scientific reasoning benchmarks from SciKnowEval
• →Text-level guidance enables use of black-box teacher models without requiring access to model logits or distributions