Evaluating LLMs on Real-World Software Performance Optimization
Researchers introduce SWE-Pro, a benchmark revealing that current Large Language Models perform poorly at real-world software performance optimization compared to expert engineers. The study shows LLMs achieve negligible runtime improvements and nearly zero memory optimizations, while human experts demonstrate 15.5x speedups and 171.3x peak memory reductions across the same tasks.