Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement

Research reveals that Large Language Models (LLMs) systematically fail at code review tasks, frequently misclassifying correct code as defective when matching implementations to natural language requirements. The study found that more detailed prompts actually increase misjudgment rates, raising concerns about LLM reliability in automated development workflows.

• →LLMs frequently misclassify correct code implementations as non-compliant when reviewing against natural language specifications.
• →More detailed prompts requiring explanations and corrections paradoxically lead to higher misjudgment rates.
• →The research exposes critical reliability issues for LLM-based code assistants in software development.
• →A Fix-guided Verification Filter is proposed to validate code using executable counterfactual evidence and benchmark tests.
• →The findings highlight the need for safeguards when integrating LLM-based reviewers in automated development pipelines.