Understanding Annotator Safety Policy with Interpretability

Researchers introduce Annotator Policy Models (APMs), interpretable machine learning models that extract and visualize annotators' implicit safety policies from labeling behavior alone. By revealing disagreement sources—operational failures, policy ambiguity, and value pluralism—APMs enable more transparent and inclusive AI safety policy design without requiring costly additional annotation.

The paper addresses a fundamental challenge in AI safety: understanding why human annotators disagree when labeling safety-sensitive content. Traditional approaches to resolving disagreement rely on asking annotators to explain their reasoning, but this method proves costly and unreliable—human introspection often misrepresents actual decision-making processes, and LLM-based explanations can be equally misleading. APMs sidestep this problem by learning annotators' implicit safety policies directly from labeling patterns, achieving over 80% accuracy in reconstructing individual decision-making frameworks.

This work emerges from a critical gap in AI governance. As language models scale, safety annotation has become industrial-scale work conducted by distributed teams with potentially conflicting interpretations of safety guidelines. Current practices treat disagreement as noise to be averaged out, missing opportunities to identify whether failures stem from genuine value differences across populations, ambiguous instructions, or quality control issues.

The research demonstrates two concrete applications with direct policy implications. First, APMs surface policy ambiguity by revealing how different annotators interpret identical safety instructions—a finding that enables clearer guideline writing. Second, they expose value pluralism across demographic groups, suggesting that monolithic safety policies may reflect majority preferences while marginalizing minority perspectives. For AI developers and safety teams, this means better policy design through data-driven insights into disagreement sources.

Looking forward, APMs could influence how AI companies structure annotation workflows and safety governance. If adopted, the methodology might shift industry practice toward more deliberative, inclusive policy-setting rather than consensus-forcing approaches. Wider adoption could also establish interpretability of annotator behavior as a best practice in model development.

• →APMs learn annotators' implicit safety policies from labeling behavior alone, achieving >80% accuracy without requiring additional annotation effort
• →The model distinguishes three sources of disagreement: operational failures, policy ambiguity, and value pluralism—each requiring different solutions
• →APMs reveal systematic safety priority differences across demographic groups, surfacing value pluralism hidden in traditional annotation approaches
• →The methodology enables more transparent and inclusive safety policy design by making annotator reasoning visible and comparable
• →Direct applications include policy clarification, quality control improvement, and deliberative incorporation of diverse perspectives in AI safety governance