When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model

A rigorous empirical study challenges claims that large language models improve hyperparameter optimization for tabular data, finding that LLM advisors' apparent advantage comes entirely from a fixed default configuration seed, not the model itself. Classical search methods with the same seed match or outperform LLM approaches within a handful of evaluations, suggesting LLM-based HPO systems offer no meaningful generalization benefit.

This research paper presents a sobering reality check for the hype surrounding LLM-powered hyperparameter optimization. The study's core finding—that LLM advisors derive their apparent strength from a pre-seeded default configuration rather than model-generated insights—exposes a methodological flaw common in prior HPO research. The researchers conducted a rigorous, budget-matched comparison across multiple benchmarks with proper statistical controls, isolating the LLM's actual contribution at just +0.40 percentage points in cross-validation accuracy, with zero improvement on held-out test sets. This matters because hyperparameter optimization directly impacts model performance, and practitioners considering LLM-based tools need accurate expectations about their utility.

The broader context reveals how LLM capabilities are often oversold in specialized domains where classical methods remain competitive. HPO has well-established solutions—Bayesian optimization, evolutionary algorithms, and random search with sensible priors—that have matured over decades. The research demonstrates that when classical methods receive equivalent seeds and budget constraints, they reach parity with LLM approaches by evaluation five and achieve substantially better performance by evaluation 12, contradicting narratives about LLMs as universally superior problem-solvers.

For the AI development community, this study provides crucial grounds for skepticism about LLM applications in technical workflows. Developers and ML engineers may waste resources implementing LLM-based HPO systems when simpler, faster alternatives deliver equivalent or superior results. The one positive finding—a rule-based confidence filter eliminating 33% of wasted compute—suggests practical value exists not in LLM reasoning but in structured filtering mechanisms.

Future work should examine whether LLMs provide advantages in high-dimensional, non-tabular spaces or when combining heterogeneous data types, while practitioners should default to classical search with domain-informed initialization rather than adopting LLM advisors based on uncontrolled comparisons.

• →LLM hyperparameter advisors' apparent superiority vanishes when classical search methods receive identical default seeds, collapsing their claimed 0.2pp lead within 5 evaluations.
• →The LLM's actual contribution is +0.40pp on cross-validation and -0.01pp on test accuracy, statistically indistinguishable from random noise.
• →Classical search with a sensible default configuration matches or exceeds LLM performance while remaining faster and computationally cheaper.
• →LLM-specific behaviors like confidence filtering offer limited value, removing 33% of compute without accuracy gains rather than improving generalization.
• →This finding applies specifically to tabular data and may not generalize to other domains, suggesting LLMs are not universally superior for technical optimization tasks.