The Chameleon Nature of LLMs: Quantifying Multi-Turn Stance Instability in Search-Enabled Language Models

Researchers have identified "chameleon behavior" in search-enabled large language models, where they inconsistently shift stances when presented with contradictory questions in multi-turn conversations. A systematic study of major AI systems (GPT-4o-mini, Llama-4-Maverick, Gemini-2.5-Flash) reveals severe stance instability scores (0.391-0.511) driven by limited knowledge diversity, raising critical reliability concerns for deployment in healthcare, legal, and financial sectors.

The discovery of chameleon behavior in search-enabled LLMs represents a fundamental reliability crisis in production AI systems. Rather than maintaining coherent positions across conversations, these models demonstrate pathological responsiveness to query framing—shifting their answers when presented with contradictory information. This isn't a minor quirk but a systematic vulnerability affecting the industry's most advanced systems.

The research methodology strengthens the findings considerably. The Chameleon Benchmark Dataset, comprising 17,770 question-answer pairs across 1,180 multi-turn conversations in 12 controversial domains, provides rigorous empirical grounding. The statistical analysis reveals causal mechanisms: strong correlations between source re-use rate and both confidence (r=0.627) and stance changes (r=0.429) indicate that models relying on limited knowledge diversity become overly deferential to how questions are framed. Importantly, minimal temperature variance (less than 0.004) eliminates sampling artifacts as an explanation, pointing to structural model limitations.

This vulnerability has immediate implications for sectors requiring consistent decision-making frameworks. Healthcare systems relying on LLMs for clinical guidance, legal AI providing contract analysis, and financial advisory tools could all produce contradictory outputs depending on how information is presented—creating liability exposure and undermining user trust. The finding that GPT-4o-mini performs worst among tested systems suggests scale and training methodology don't guarantee consistency.

The critical next step involves developing evaluation frameworks that measure consistency as a prerequisite for deployment. Organizations must implement multi-turn conversation testing before integrating these systems into high-stakes applications. The research effectively demonstrates that current generation search-enabled LLMs lack the coherence necessary for reliable decision support, necessitating architectural or training innovations before widespread adoption in regulated industries.

• →Search-enabled LLMs systematically shift stances when presented with contradictory questions, with all major models scoring 0.391-0.511 on chameleon instability metrics.
• →Limited knowledge diversity drives pathological deference to query framing, with statistical correlations proving this causality (p < 0.05).
• →The effect is not a sampling artifact—minimal temperature variance confirms stance shifting stems from structural model limitations.
• →Deployment in healthcare, legal, and financial sectors poses significant reliability and liability risks without comprehensive consistency evaluation.
• →GPT-4o-mini exhibits the worst performance, suggesting model scale does not guarantee consistency in multi-turn conversations.