FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics

Researchers introduce FML-Bench, a standardized benchmark for evaluating AI research agents that separates strategy from infrastructure, revealing that simple greedy algorithms perform comparably to complex tree-search methods. The study identifies that exploration strategy effectiveness depends on the underlying structure of optimization opportunities, with an adaptive agent demonstrating superior performance by switching strategies based on improvement stagnation detection.

FML-Bench addresses a critical gap in AI agent evaluation methodology. Previous benchmarks conflated algorithmic strategy with implementation details, making it impossible to isolate which strategic choices actually drive performance. By controlling for execution infrastructure while measuring 12 process-level behavioral metrics across 18 ML tasks, the researchers enable meaningful comparison of fundamentally different search approaches.

The counterintuitive finding that greedy hill-climbing nearly matches tree-search performance challenges assumptions about algorithm complexity. This suggests that agent effectiveness isn't determined by sophistication alone but by alignment between strategy and problem structure. The framework reveals that greedy methods excel when improvement opportunities are densely distributed, while tree-search and evolutionary algorithms perform better in sparse opportunity landscapes. This insight proves valuable for researchers designing agents for specific domains.

The adaptive agent's superior performance by detecting stagnation and switching exploration modes validates a hybrid approach, demonstrating practical applicability of the benchmark's insights. The discovery that early convergence and focused exploration predict success while solution diversity and computational cost do not reshapes conventional optimization wisdom for AI research tasks.

For the broader AI research community, FML-Bench provides essential infrastructure for systematic agent development. Rather than evaluating agents in isolation on disparate benchmarks, researchers now have a controlled environment to test strategic hypotheses. This accelerates the meta-science of AI research automation, potentially enabling more efficient ML advancement cycles.

• →Simple greedy hill-climbing nearly matches complex tree-search performance on ML research tasks, challenging assumptions about algorithm sophistication
• →Agent effectiveness depends on alignment between search strategy and opportunity structure density rather than complexity alone
• →Early convergence and directionally focused exploration are the strongest predictors of final performance in AI research agents
• →An adaptive agent that switches exploration strategies upon detecting stagnation outperforms six representative baseline approaches
• →FML-Bench enables controlled evaluation by separating agent strategy from execution infrastructure using 12 process-level behavioral metrics