Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews

Researchers developed a method to detect AI-generated content at scale and found that 6.5-16.9% of peer reviews at major AI conferences after ChatGPT's release were substantially modified by LLMs. The study reveals concerning patterns where AI-generated reviews correlate with lower reviewer confidence, last-minute submissions, and reduced engagement in rebuttals.