Digital Skin, Digital Bias: Uncovering Tone-Based Biases in LLMs and Emoji Embeddings

Researchers conducted the first large-scale study comparing bias in skin-toned emoji representations across specialized emoji models and four major LLMs (Llama, Gemma, Qwen, Mistral), finding that while LLMs handle skin tone modifiers well, popular emoji embedding models exhibit severe deficiencies and systemic biases in sentiment and meaning across different skin tones.

This research addresses a critical blind spot in AI fairness: the representational bias embedded in how models interpret skin-toned emojis, which have become essential markers of identity and inclusion in digital communication. While much attention focuses on text-based bias in LLMs, this study reveals that foundational emoji models—tools widely deployed across platforms—systematically misrepresent skin tones through skewed sentiment associations and inconsistent semantic meanings. The findings are particularly concerning because emoji embedding models like emoji2vec and emoji-sw2v are specialized systems expected to handle these symbols accurately, yet they underperform compared to general-purpose LLMs. This represents a gap between expectations and reality in AI safety. The broader context reflects growing awareness that biases pervade multiple layers of AI infrastructure, not just training data or alignment. For platform developers and AI practitioners, the study signals an urgent audit requirement: systems mediating human communication must actively measure and correct representational harms, especially for features explicitly designed to foster inclusion. The research underscores that equity in AI isn't merely an ethical aspiration but a functional requirement, as biased emoji representation could subtly reinforce social hierarchies at scale. Looking forward, the industry should establish standardized bias evaluation frameworks for emoji and symbol representations, integrate tone-awareness testing into model evaluation pipelines, and prioritize fixing existing models already deployed across major platforms.

• →Specialized emoji models show severe deficiencies in handling skin-toned emojis while general LLMs demonstrate robust support for skin tone modifiers.
• →Systemic biases manifest as skewed sentiment polarity and inconsistent meanings associated with the same emoji across different skin tones.
• →Current AI safety practices overlook representational harms in foundational models that mediate digital communication and identity expression.
• →Platforms using emoji embeddings risk perpetuating societal biases through subtle, systemic disparities in how different groups are represented.
• →Urgent need for standardized bias auditing frameworks and mitigation strategies before deploying emoji representation systems at scale.