GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization

Researchers demonstrate that large language models can effectively forecast GPU kernel performance, reducing expensive on-device evaluations during optimization searches. By acting as selective surrogates that know their confidence limits, LLMs enable kernel searches to evaluate multiple candidates under fixed GPU budgets, ultimately discovering faster kernels than baseline approaches.

This research addresses a critical bottleneck in GPU kernel optimization where repeated hardware measurements consume significant computational resources. As large language models increasingly generate novel kernels and evolutionary search scales to larger candidate pools, the cost of validating each proposal on actual GPUs becomes prohibitively expensive. The study proposes using LLMs as performance predictors—virtual GPU surrogates that forecast kernel runtime before costly compilation and execution occur.

The breakthrough lies not merely in prediction accuracy but in selective confidence calibration. The LLMs learn when to defer to actual GPU measurement rather than trusting inaccurate forecasts, creating a hybrid evaluation system that maximizes search efficiency. Through reinforcement learning optimization, the researchers improved both forecast precision and confidence estimates, enabling searches to explore several times more candidates within identical GPU budgets.

This advancement has profound implications for deep learning infrastructure development. GPU kernel optimization directly impacts the efficiency and cost of training and inference for AI models. By reducing measurement overhead, researchers and engineers can iterate faster on kernel designs, potentially leading to broader accessibility of GPU optimization techniques previously limited to well-resourced organizations. The ability to evaluate more candidates within fixed compute budgets fundamentally changes the economics of kernel search, making optimization more practical across industry and academic settings.

Future research will likely explore extending this approach to other hardware optimization domains and integrating LLM surrogates into production kernel-search pipelines. The framework suggests LLMs possess latent understanding of computational performance characteristics that extends beyond code generation.

• →LLMs can accurately forecast GPU kernel performance when trained with confidence calibration to know their limitations.
• →Selective surrogates that defer uncertain predictions to actual GPU measurement enable more efficient search under limited budgets.
• →Reinforcement learning improves both forecast accuracy and confidence calibration of LLM-based performance predictors.
• →Kernel searches using LLM surrogates evaluate several times more candidates while discovering faster kernels than baseline approaches.
• →This approach positions LLMs as virtual GPU models rather than solely as code generation tools for optimization tasks.