Cognitive models can reveal interpretable value trade-offs in language models

Researchers developed a framework using cognitive models from psychology to analyze value trade-offs in language models, revealing how AI systems balance competing priorities like politeness and directness. The study shows LLMs' behavioral profiles shift predictably when prompted to prioritize certain goals and are influenced by reasoning budgets and training dynamics.

• →Cognitive models can systematically evaluate alignment-relevant trade-offs in language models by modeling competing utility functions.
• →LLMs' behavioral profiles shift predictably when prompted to prioritize specific goals and are amplified by small reasoning budgets.
• →Post-training dynamics show large shifts in AI values early in training with persistent effects from base model and pretraining data choices.
• →The framework can diagnose social behaviors like sycophancy in AI systems beyond just polite speech patterns.
• →Base model and pretraining data have more influence on AI behavior than feedback dataset or specific alignment methods.