#evaluation-gaps News & Analysis

2 articles tagged with #evaluation-gaps. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBearisharXiv – CS AI · Jun 27/10

🧠

Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations

A comprehensive study examining 186 first-party AI model evaluation reports and 248 third-party sources reveals significant gaps in social impact assessments. Developers consistently under-report on bias, environmental costs, and labor impacts, while only they can authoritatively disclose data provenance and infrastructure details—information often withheld unless tied to compliance or product adoption.

AINeutralarXiv – CS AI · Apr 146/10

🧠

Principles Do Not Apply Themselves: A Hermeneutic Perspective on AI Alignment

A new arXiv paper argues that AI alignment cannot rely solely on stated principles because their real-world application requires contextual judgment and interpretation. The research shows that a significant portion of preference-labeling data involves principle conflicts or indifference, meaning principles alone cannot determine decisions—and these interpretive choices often emerge only during model deployment rather than in training data.