IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages

Researchers introduced IndoBias, a benchmark specifically designed to evaluate bias in Large Language Models across Indonesian and three local languages (Javanese, Sundanese, Makasar). The study reveals that existing LLMs exhibit significant bias toward prototypical Indonesian sentences and particularly strong bias in local languages regarding ideology and religion, highlighting the critical gap in bias research for culturally and linguistically diverse contexts.

The IndoBias benchmark addresses a substantial oversight in AI bias research by focusing on Indonesia's exceptionally complex linguistic and cultural landscape. With over 1,300 ethnic groups and 700 indigenous languages, Indonesia represents a unique testing ground for evaluating representational fairness that most mainstream LLM bias studies overlook. This research matters because bias in language models directly impacts millions of users across underrepresented regions, potentially perpetuating harmful stereotypes at scale.

The study employs a dual-track evaluation methodology combining contrastive pairs with generation-based approaches grounded in established social science frameworks. Results demonstrate that decoder-based models show pronounced bias toward prototypical Indonesian text, while local languages face disproportionate bias in ideology and religion categories. The non-uniform stereotype polarity across local entities suggests bias patterns don't operate uniformly even within a single country.

A critical finding concerns data provenance: Common Crawl texts introduce significantly more bias during pretraining compared to human-curated sources like Wikipedia and news articles. Conversely, incorporating local language data generally increases bias rather than mitigating it, suggesting that simply adding diverse language data without careful curation may amplify rather than solve representation problems.

For AI developers and technology companies targeting Southeast Asian markets, this research indicates the need for culturally-contextualized bias evaluation frameworks rather than universal standards. Organizations building LLMs for Indonesian-speaking regions must implement localized debiasing strategies and reconsider training data composition. This work establishes a methodological blueprint for similar research in other culturally diverse, multilingual regions globally.

• →Existing LLMs exhibit strong bias toward prototypical Indonesian sentences while local languages face heightened bias in ideology and religion categories.
• →Common Crawl pretraining data introduces significantly more bias than human-curated sources like Wikipedia and news articles.
• →Adding local language data to pretraining generally increases bias rather than reducing it, indicating that data quantity alone doesn't ensure fairness.
• →LLMs display non-uniform stereotype polarity when responding to different local entities, suggesting context-dependent bias patterns.
• →Culture-specific bias evaluation frameworks are essential for accurately assessing fairness in linguistically and ethnically diverse regions.