AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference

AgentCompile is an LLM-guided CUDA inference compiler that uses large language models to optimize transformer model execution on GPUs. The system achieves 4-5.66x speedup over PyTorch across popular models like Qwen and Llama through intelligent specialization decisions and empirical validation.

AgentCompile addresses a critical bottleneck in modern AI deployment: the compilation and optimization of transformer inference on GPUs. While transformer models have become ubiquitous, converting model graphs into efficient CUDA kernels remains a complex engineering challenge requiring semantic understanding of where specialization provides genuine benefits. This research tackles that gap by leveraging LLMs not as execution engines but as intelligent metadata providers, fundamentally shifting how the AI infrastructure stack approaches optimization.

The approach reflects broader maturation in AI systems engineering. Rather than treating LLMs as replacements for traditional compilation logic, AgentCompile uses them where they excel—understanding semantic relationships and proposing candidates—while preserving compiler rigor through empirical validation and fallback mechanisms. This hybrid methodology mirrors real-world infrastructure decisions where multiple specialized components outperform monolithic approaches.

For the inference optimization market, these results carry significant implications. Achieving 4-5.66x speedups directly impacts deployment costs, latency-sensitive applications, and computational efficiency. Organizations running inference-heavy workloads could substantially reduce infrastructure spending or serve more users with existing hardware. The open-source commitment signals confidence in the approach and could accelerate adoption across the developer ecosystem.

The work establishes a template for combining LLM-based semantic reasoning with compiler infrastructure. Success here may influence how downstream inference optimization tools integrate language models, shifting from simple code generation toward sophisticated decision support systems. As inference becomes increasingly cost-critical for LLM application providers, such compilation improvements represent meaningful competitive advantages.

• →AgentCompile achieves 4-5.66x speedup on Qwen and Llama models through LLM-guided CUDA specialization with empirical validation
• →LLMs function as semantic metadata providers rather than direct execution engines, improving decision-making while maintaining compiler safety guarantees
• →The hybrid approach includes fallback mechanisms to prevent unprofitable or unsupported optimizations from degrading performance
• →Open-source release indicates broad applicability and potential for rapid ecosystem adoption in inference optimization
• →Results demonstrate meaningful cost and efficiency improvements for production inference workloads across representative model architectures