Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Study

Researchers evaluated four AI Ethics Tools (AIETs) applied to Portuguese language models through interviews with 35 developers, finding that while these tools provide general ethical guidance, they fail to address language-specific nuances and cannot effectively identify potential harms in non-English models.

This research exposes a critical gap in current AI ethics infrastructure that extends beyond academic curiosity into practical deployment challenges. The study evaluated Model Cards, ALTAI, FactSheets, and Harms Modeling against Portuguese language models—a methodologically sound approach that reveals how ethics frameworks designed for English-centric AI systems falter when applied to other languages and cultural contexts. The findings suggest that widely-adopted AI ethics tools, while providing structural guidance on documentation and transparency, operate at too abstract a level to catch language-specific harms and biases.

The broader context shows AI ethics has evolved from theoretical frameworks into standardized toolkits that organizations now widely implement. However, this research demonstrates the tools' effectiveness remains unvalidated in real-world developer workflows, particularly for non-English markets. The 213-tool landscape review followed by selective evaluation of just four tools indicates fragmentation and inconsistent quality across the field.

For the AI industry, this creates significant liability exposure. As companies deploy language models globally, ethics tools that fail to identify language-specific harms represent regulatory and reputational risks. Developers face a dilemma: these tools provide the appearance of responsible AI development while potentially masking actual ethical problems. This gap is especially concerning for emerging markets where Portuguese, along with other non-English languages, drives substantial commercial AI deployment.

The research suggests future AI ethics tools must incorporate multilingual and culturally-contextualized evaluation frameworks rather than relying on English-derived methodologies. Organizations deploying language models in non-English markets cannot rely solely on existing ethics tools and need supplementary assessment mechanisms tailored to specific linguistic and cultural contexts.

• →Existing AI ethics tools provide general guidance but fail to detect language-specific harms in non-English models like Portuguese language systems.
• →Four major ethics tools (Model Cards, ALTAI, FactSheets, Harms Modeling) lack effectiveness in addressing idiomatic expressions and cultural nuances.
• →The field contains 213+ AI ethics tools with highly variable documentation and unproven real-world utility for developers.
• →Companies deploying global language models face regulatory risk by relying on ethics frameworks not validated across linguistic contexts.
• →Future ethics infrastructure must incorporate multilingual assessment methodologies rather than generalizing from English-centric approaches.