LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy

Researchers applied Signal Detection Theory to analyze three large language models across 168,000 trials, finding that temperature parameter changes both sensitivity and response bias simultaneously. The study reveals that traditional calibration metrics miss important diagnostic information that SDT's full parametric framework can provide.

• →Temperature parameter in LLMs functions differently than expected, changing both the answer generation and confidence levels rather than just confidence.
• →All tested models showed unequal-variance evidence distributions with instruction-tuned models exhibiting more extreme asymmetry than base models.
• →Traditional calibration metrics like Expected Calibration Error conflate sensitivity and bias components that Signal Detection Theory can separate.
• →Models with similar calibration scores can occupy very different positions in sensitivity-bias space, making SDT analysis more informative.
• →The research provides a new framework for evaluating LLM performance beyond standard calibration metrics.