SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

Researchers introduce SWE-CI, a new benchmark that evaluates AI agents' ability to maintain codebases over time through continuous integration processes. Unlike existing static bug-fixing benchmarks, SWE-CI tests agents across 100 long-term tasks spanning an average of 233 days and 71 commits each.

• →SWE-CI is the first repository-level benchmark built on continuous integration loops for evaluating AI coding agents.
• →The benchmark shifts evaluation from static functional correctness to dynamic long-term code maintainability.
• →Each of the 100 tasks represents real-world evolution spanning an average of 233 days and 71 consecutive commits.
• →The benchmark requires agents to perform dozens of rounds of analysis and coding iterations systematically.
• →This addresses limitations of existing benchmarks like SWE-bench that focus only on one-shot static repairs.