Same Words, Different Judgments: Modality Effects on Preference Alignment

Researchers conducted a cross-modal study comparing human preference annotations between text and audio formats for AI alignment. The study found that while audio preferences are as reliable as text, different modalities lead to different judgment patterns, with synthetic ratings showing promise as replacements for human annotations.

• →Audio preferences prove as reliable as text preferences with good inter-rater agreement at approximately 9 raters.
• →Audio evaluators show narrower decision thresholds and reduced length bias compared to text evaluators.
• →Cross-modality agreement between text and audio preferences is near-chance level despite identical semantic content.
• →Synthetic ratings align well with human judgments and can predict inter-rater agreement.
• →This represents the first ICC-based reliability characterization in preference annotation literature for both modalities.