Provable and Practical In-Context Policy Optimization for Self-Improvement

Researchers introduce In-Context Policy Optimization (ICPO), a new method that allows AI models to improve their responses during inference through multi-round self-reflection without parameter updates. The practical ME-ICPO algorithm demonstrates competitive performance on mathematical reasoning tasks while maintaining affordable inference costs.

• →ICPO enables AI models to optimize responses at test-time using self-assessed rewards without changing model parameters.
• →Theoretical framework proves that linear self-attention models can imitate policy optimization algorithms under specific training conditions.
• →ME-ICPO uses minimum entropy selection and majority voting to ensure robust self-assessment of response quality.
• →The method achieves top-tier performance on mathematical reasoning benchmarks while keeping inference costs manageable.
• →ICPO provides a principled theoretical understanding of self-reflection mechanisms in large language models.