#benchmark-contamination News & Analysis

4 articles tagged with #benchmark-contamination. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

4 articles

AIBearisharXiv – CS AI · Apr 147/10

🧠

Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards

Researchers identify systematic measurement flaws in reinforcement learning with verifiable rewards (RLVR) studies, revealing that widely reported performance gains are often inflated by budget mismatches, data contamination, and calibration drift rather than genuine capability improvements. The paper proposes rigorous evaluation standards to properly assess RLVR effectiveness in AI development.

AIBearisharXiv – CS AI · Mar 37/103

🧠

On The Fragility of Benchmark Contamination Detection in Reasoning Models

New research reveals that benchmark contamination in language reasoning models (LRMs) is extremely difficult to detect, allowing developers to easily inflate performance scores on public leaderboards. The study shows that reinforcement learning methods like GRPO and PPO can effectively conceal contamination signals, undermining the integrity of AI model evaluations.

$NEAR

AINeutralarXiv – CS AI · Apr 146/10

🧠

Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs

Researchers introduce a multi-agent framework to map data lineage in large language models, revealing how post-training datasets evolve and interconnect. The analysis uncovers structural redundancy, benchmark contamination propagation, and proposes lineage-aware dataset construction to improve LLM training diversity and quality.

AINeutralOpenAI News · Feb 236/105

🧠

Why we no longer evaluate SWE-bench Verified

SWE-bench Verified, a popular coding evaluation benchmark, is being discontinued due to increasing contamination and flawed testing methodology. The analysis reveals training data leakage and unreliable test cases that fail to accurately measure AI coding capabilities, with SWE-bench Pro recommended as the replacement.