Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning

Researchers introduce the Triangulated Preference Shift score, an automated metric that identifies lexical biases introduced during preference learning stages (like RLHF) in large language models without requiring manual curation. The metric isolates language pattern shifts across six model families, revealing that preference tuning may push models toward a 'language of prestige' that diverges from natural human language usage.

Large language models undergo systematic behavioral changes during preference-learning stages, where reinforcement learning from human feedback and similar techniques optimize for human preferences. While these methods generally improve model utility, they simultaneously introduce measurable lexical biases—overuse of certain words like 'delve' or 'furthermore' and preferences for specific formatting patterns that don't naturally emerge from base models. Previous research on this phenomenon relied heavily on manual curation, creating bottlenecks in scale and introducing subjective bias into the measurement process itself.

This research addresses a critical gap by proposing an automated, curation-free approach that triangulates between three reference points: human gold standards, base model outputs, and instruction-tuned variants. This methodological innovation enables researchers to isolate behavioral shifts caused specifically by preference tuning rather than base model training. Testing across six model families provides empirical breadth and grounds findings in measurable evidence rather than anecdotal observation.

The implications extend beyond academic curiosity into model development and alignment. If preference learning systematically shifts models toward artificial 'prestige' language patterns, this raises questions about whether alignment processes inadvertently create unrealistic communication styles that may reduce user trust or practical utility. Understanding these systematic biases helps AI developers make informed choices during training, potentially improving alignment quality.

The Triangulated Preference Shift score establishes an automated foundation for ongoing monitoring of model behavior changes. Future work could apply this metric to detect problematic shifts early in development cycles, enabling course corrections before deployment. This contributes to broader trustworthy AI objectives by making preference-tuning effects transparent and measurable.

• →A new automated metric quantifies lexical biases introduced specifically during preference learning without manual curation.
• →Preference tuning systematically shifts model language toward patterns labeled a 'language of prestige' that diverges from natural human speech.
• →Testing across six model families reveals this bias is consistent and measurable across different LLM architectures.
• →Automated bias detection enables earlier intervention during model development, supporting better alignment practices.
• →The research methodology establishes a scalable framework for monitoring behavioral shifts in AI systems over time.