Annotator Positionality as Signal: Psychometric Weighting for Anti-Autistic Ableism Detection

Researchers developed a bias-aware evaluation framework to detect anti-autistic ableism in large language models, using psychometrically-weighted annotations from autistic community members as ground truth. The study reveals that LLMs frequently produce harmful outputs, misclassify community language, and rely on surface-level keyword matching rather than contextual understanding of speaker identity and intent.

This research addresses a critical gap in AI safety and fairness by examining how large language models understand and reproduce ableist bias specifically targeting autistic communities. The study's innovation lies in its methodological approach: rather than using conventional majority-vote aggregation, the researchers weighted annotations based on psychometric validity and annotator positionality, giving greater influence to perspectives from autistic individuals and autism-accepting experts. This represents a meaningful departure from standard practices that often dilute marginalized community perspectives.

The findings expose fundamental limitations in how current LLMs process nuanced language around disability. Models trained on broad internet data appear to conflate keyword presence with harm, failing to recognize contextual factors like whether language fosters community solidarity or inflicts external harm. When evaluation prompts are masked—hiding information about content type—models express more negative attitudes toward autistic people, suggesting their bias is partially obscured by surface-level framing rather than genuinely understood.

For AI developers and deployment stakeholders, this research demonstrates that standard benchmarking approaches inadequately capture bias in high-stakes decision-making systems. LLMs increasingly influence content moderation, hiring, healthcare recommendations, and educational settings where such failures carry real consequences for vulnerable populations. The study's framework offers a replicable methodology for other marginalized communities seeking more representative AI evaluation standards.

Organizations deploying LLMs in consequential domains should recognize that conventional evaluation methods mask harmful behaviors. Future work likely involves expanding similar community-proximate evaluation frameworks across disabilities and other marginalized identities, potentially reshaping how industry standards for AI fairness are established.

• →LLMs frequently produce harmful outputs toward autistic people and misclassify reclaimed community language as ableist
• →Standard majority-vote annotation aggregation systematically underweights perspectives from autistic and autism-accepting annotators
• →Models rely on surface-level keyword matching rather than understanding context, speaker identity, and whether language causes in-group or out-group harm
• →Masking assessment instruments reveals stronger bias, suggesting models' harmful outputs are partially obscured in standard evaluation settings
• →Community-proximate, psychometrically-weighted evaluation frameworks provide stricter standards for detecting marginalized group bias in AI systems