Social Bias in LLM-Generated Code: Benchmark and Mitigation

Researchers have identified severe social bias in code generated by large language models, with bias scores reaching 60.58% across four major models. They propose a Fairness Monitor Agent that reduces bias by 65.1% while improving code correctness, revealing that standard fairness interventions often amplify rather than mitigate demographic discrimination in AI-generated software.

The emergence of social bias in LLM-generated code represents a critical gap between technical capability and ethical deployment. While LLMs excel at functional correctness, they systematically embed demographic discrimination into applications that directly affect users—a problem largely invisible in current evaluation frameworks that prioritize only whether code works, not whom it harms. This research exposes a troubling pattern: naive attempts to address fairness through prompt engineering and explicit fairness instructions backfire, suggesting that bias mitigation requires architectural solutions rather than surface-level interventions.

This work builds on growing recognition that AI systems encode societal biases at scale. Unlike previous studies focusing on model outputs, examining bias in code generation matters because it affects real software deployed across hiring systems, lending platforms, and content moderation tools. The 343-task benchmark spanning seven demographic dimensions provides the most comprehensive evaluation to date of how code generation models handle fairness considerations.

The proposed Fairness Monitor Agent demonstrates that modular, task-aware auditing during code generation outperforms both no intervention and aggressive fairness instructions. By analyzing task descriptions to determine which attributes should be restricted, FMA achieves 65.1% bias reduction while simultaneously improving functional correctness from 75.80% to 83.97%—proving that fairness and functionality are not opposing goals. This finding reshapes developer priorities: integrating fairness checks into generation pipelines becomes economically rational, not merely ethical.

Looking ahead, enterprises deploying LLM-based code generation face pressure to implement bias detection frameworks. The research suggests that post-hoc auditing using modular agents offers practical adoption pathways without requiring complete pipeline redesigns, likely influencing tool development and procurement decisions across enterprise software development.

• →LLMs generate code with bias scores up to 60.58%, indicating severe demographic fairness issues across all major models studied.
• →Standard fairness interventions like chain-of-thought reasoning and fairness personas actually amplify bias rather than reduce it.
• →The Fairness Monitor Agent reduces bias by 65.1% and improves code correctness simultaneously, proving fairness and functionality are complementary goals.
• →Modular auditing approaches outperform explicit fairness instructions to all agent roles, suggesting diffused responsibility dilutes impact.
• →Code generation evaluation frameworks must expand beyond functional correctness to assess demographic fairness across seven dimensions.