Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents

Researchers investigate whether large language model agents actually follow their stated reasoning when making decisions, using a Texas Poker simulator as a controlled test environment. The study identifies a 'faithfulness gap' by decomposing agent behavior into two distinct steps—reasoning-to-conclusion and conclusion-to-action—revealing they behave oppositely, raising concerns about LLM reliability in applications requiring transparent decision-making.

This research addresses a fundamental trust problem with language model agents: the disconnect between their stated reasoning and actual behavior. While LLMs can articulate logical arguments convincingly, they may not actually implement those conclusions in subsequent actions. The study's use of Texas Poker provides an elegant experimental framework because each decision point has a mathematically verifiable optimal action, eliminating ambiguity about what constitutes correct behavior.

The finding that reasoning-conclusion and conclusion-action steps behave oppositely suggests the fidelity problem is more nuanced than simple failure. An LLM might correctly reason through a problem but fail to execute, or conversely, produce flawed reasoning while still arriving at appropriate actions. This asymmetry is critical for understanding where interventions should focus.

For developers deploying LLM agents in consequential domains—financial analysis, medical diagnosis, legal reasoning—the implications are substantial. If agents cannot reliably execute their stated reasoning, traditional validation methods that examine reasoning traces become unreliable quality controls. Users cannot trust that observing good reasoning guarantees good outcomes. This affects system design decisions around oversight mechanisms and human-in-the-loop safeguards.

The research points toward a need for dual-layer verification systems that independently validate both reasoning quality and action fidelity rather than assuming they correlate. Going forward, developers must characterize these failure modes empirically for their specific use cases rather than assuming general-purpose LLM agents will reliably operate as described.

• →LLM agents exhibit a 'faithfulness gap' where stated reasoning does not reliably predict subsequent actions
• →Reasoning-to-conclusion and conclusion-to-action failures occur independently and oppositely in language models
• →Traditional reasoning verification methods may provide false confidence about agent reliability
• →Texas Poker simulator environment enables verifiable measurement of agent fidelity through optimal reference actions
• →Dual-layer verification systems are needed to validate both reasoning quality and action execution separately