Does Topic Sentiment Cause Perceived Ideology? Comparing Human and LLM Annotations in Political News Articles

A research study compares how human annotators and large language models (GPT-4o-mini, Llama-3.3-70B) assign political ideology labels to news articles, finding that fine-tuned GPT-4o-mini models develop spurious correlations between sentiment and ideology that don't exist in human judgment. This reveals a critical vulnerability in using LLM annotations as training data for downstream tasks.

This study addresses a fundamental problem in AI-assisted research: whether machine learning models trained on human-labeled data actually replicate human reasoning or instead learn statistical shortcuts invisible to standard evaluation metrics. Researchers analyzed political news articles using causal inference techniques to determine whether topic sentiment meaningfully influences ideology perception.

The findings diverge sharply between human and machine annotators. Human experts show no significant causal relationship between sentiment and ideology labels—suggesting they evaluate political slant based on substantive content rather than emotional tone. Fine-tuned GPT-4o-mini, however, demonstrates strong spurious coupling between sentiment and ideology, despite achieving the highest classification accuracy (F1=72.48). This disconnect reveals shortcut learning: the model internalized a sentiment-ideology correlation present in its training data that doesn't reflect actual human judgment patterns.

This has significant implications for AI development pipelines. Organizations increasingly rely on LLM annotations as "silver labels" to reduce human annotation costs, then use these labels to train downstream models. If the LLM captures spurious correlations invisible to standard metrics, this contamination propagates through the entire pipeline. A model achieving high F1 scores may still encode fundamental misunderstandings of the task domain.

The research underscores why evaluation metrics alone cannot guarantee model alignment with human reasoning. Future work should incorporate causal analysis and domain-specific validation beyond classification accuracy, particularly for tasks involving subjective judgment like ideology assessment.

• →Fine-tuned language models can achieve high accuracy while learning spurious correlations between features that humans don't use for judgment
• →Standard F1 metrics fail to detect shortcut learning that undermines causal validity in downstream applications
• →Using LLM annotations as training data without validation can propagate model biases through entire ML pipelines
• →Causal inference methods reveal annotation quality issues invisible to traditional classification metrics
• →Human and machine annotators process information fundamentally differently despite similar task performance