AINeutralarXiv – CS AI · 14h ago6/10
🧠
Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA
Researchers introduce Code-QA-Bench, an automated framework that generates repository-level code understanding benchmarks while distinguishing genuine code comprehension from documentation recall. Testing four frontier AI models reveals that code access is the primary driver of performance, while documentation provides marginal benefits, suggesting current models excel at code reasoning when source material is available.