Regression Language Models for Code

Researchers have developed Regression Language Models (RLMs) that use frozen LLM encoders to predict numeric code execution outcomes across multiple programming languages and domains. A 300M parameter model demonstrates strong performance predicting memory footprint, GPU latency, neural network accuracy, and hardware platform performance without domain-specific feature engineering.

The emergence of unified regression language models represents a significant shift in how systems approach code performance prediction. Traditionally, predicting numeric outcomes from code—such as memory usage, execution latency, or model accuracy—required extensive domain-specific feature engineering and specialized tools for different programming languages. This new approach leverages frozen LLM encoders to directly process raw code text, eliminating the need for hand-crafted features while maintaining competitive performance across diverse prediction tasks.

The technical achievement demonstrates that general-purpose language models contain implicit knowledge about code behavior and computational complexity. By fine-tuning regression heads on frozen encoders rather than retraining entire models, the approach achieves computational efficiency while maintaining flexibility. The 300M parameter RLM's performance across 17 programming languages and its competitive results on neural architecture search spaces suggest that code understanding has become sufficiently generalizable that one unified model can outperform specialized, task-specific systems.

For the software development ecosystem, this enables developers to make performance predictions earlier in the coding process without requiring specialized profiling tools or expert knowledge. Data scientists and ML engineers benefit from automated architecture search capabilities that match or exceed graph neural network approaches. The technology reduces barriers to performance optimization by democratizing access to prediction capabilities previously requiring manual analysis or expensive computational profiling.

The implications extend to automated code optimization, resource allocation in cloud computing, and real-time performance estimation during development. As these models scale and incorporate more training data, they may become integral components in IDE tooling, continuous integration pipelines, and automated system design workflows.

• →Unified RLMs eliminate the need for domain-specific feature engineering across multiple code prediction tasks
• →A 300M parameter model achieves >0.9 Spearman-rank correlation on competitive programming benchmarks
• →Single model predicts performance metrics across memory footprint, GPU latency, neural network accuracy, and hardware platforms
• →RLM achieves competitive performance with graph neural networks on neural architecture search design spaces
• →Frozen LLM encoders enable efficient transfer learning without full model retraining