Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces

Researchers introduce OPT*, a scalable benchmark for training large language models to perform step-by-step optimization reasoning across expanding search spaces. The framework combines feasibility checkers with complexity parameters that scale task difficulty without requiring new human labels, enabling both solver-guided and offline reinforcement learning approaches to improve LLM reasoning capabilities.

OPT* addresses a critical gap in LLM training methodologies by extending beyond mathematical and coding domains into broader optimization problems that require finding high-value feasible solutions among multiple valid alternatives. This work builds on recent advances in verifiable reward training but recognizes that real-world decision-making often involves navigating complex trade-offs and constraints that current benchmarks inadequately capture. The framework's elegant design—using complexity parameters to expand search spaces without requiring additional human annotation—makes it practical for large-scale training while maintaining rigorous evaluation standards.

The research proposes two distinct training regimes reflecting real-world constraints. Solver-guided online policy optimization leverages external solvers as value oracles, applying rank-based reward shaping to reinforce superior decision paths. Conversely, search-based offline RL accommodates scenarios where solvers are unavailable, broadening the framework's applicability. The theoretical contribution linking success to information extraction per unit of search budget provides principled guidance for understanding optimization efficiency.

For the AI development community, OPT* offers both a standardized evaluation metric and a practical training methodology that could accelerate progress toward more capable reasoning systems. The benchmark's scalability addresses a persistent challenge in LLM evaluation—creating difficulty gradients that stretch models without constantly generating new training data. Early empirical results demonstrating improved step-by-step reasoning suggest the approach effectively translates to enhanced performance on optimization-like tasks. This work has implications for enterprise applications requiring planning, resource allocation, and constraint satisfaction, where LLMs currently struggle with complex sequential decisions.

• →OPT* enables scalable training of LLM optimization reasoning by using complexity parameters that expand search spaces without requiring new human labels.
• →The framework supports both solver-guided online policy optimization and search-based offline RL, accommodating different resource constraints.
• →Theoretical analysis connects search efficiency to information extraction per unit of budget, providing principled guidance for optimization tasks.
• →Training on OPT* demonstrably improves step-by-step optimization-like reasoning beyond mathematical and coding domain capabilities.
• →The benchmark's design addresses a critical gap in LLM evaluation by capturing real-world decision-making across expanding feasible solution spaces.