Multi-Stakeholder LLM Alignment: Decomposing Estimation from Aggregation

Researchers present DecompR, a method to improve how large language models handle tasks with conflicting stakeholder preferences by separating utility estimation from aggregation. Traditional holistic LLM judges create unstable implicit weights that cause significant score variability, especially as stakeholder numbers increase; the proposed approach fixes weights based on query structure before scoring to eliminate candidate-dependent weight drift.

This academic research addresses a fundamental challenge in deploying large language models for real-world applications where multiple parties have competing interests. Current LLM evaluation methods struggle because they attempt to estimate and aggregate stakeholder utilities simultaneously, creating what researchers term 'weighting noise'—unpredictable shifts in scores that grow more pronounced as the number of stakeholders increases. This instability undermines confidence in model outputs for applications ranging from content moderation to recommendation systems.

The problem emerges from how traditional approaches conflate two distinct tasks: measuring individual satisfaction and combining those measurements into a final score. When an LLM judge tries to do both at once during candidate evaluation, the weights assigned to different stakeholders become implicitly dependent on the specific candidate being scored, introducing systematic bias. This is particularly problematic in scenarios with dispersed preferences where stakeholders want fundamentally different outcomes.

DecompR tackles this by separating concerns through counterfactual calibration. Weights are determined upfront based on the query structure and stakeholder roles, then held constant while individual utilities are estimated independently for each candidate. This architectural change removes the candidate-dependent drift that plagues existing methods and reduces overall estimation noise.

For organizations deploying LLMs in multi-stakeholder environments, this research validates their concerns about evaluation stability and provides a technical pathway toward more reliable outputs. The approach has implications for enterprise applications, governance systems, and any deployment requiring balanced consideration of diverse preferences. As LLMs become more central to consequential decision-making, solving the multi-stakeholder alignment problem becomes increasingly critical for user trust and regulatory compliance.

• →Traditional LLM judges create unstable weights when balancing conflicting stakeholder preferences, causing significant score variability
• →Weighting noise increases proportionally with stakeholder count, making systems less reliable as complexity grows
• →DecompR separates utility estimation from aggregation by fixing weights before candidate scoring based on query structure
• →The method eliminates candidate-dependent weight drift and reduces estimation noise in multi-stakeholder evaluation tasks
• →Stable multi-stakeholder LLM alignment is essential for enterprise applications and governance systems relying on diverse preference balancing