AINeutralarXiv – CS AI · 6d ago7/10
🧠Researchers investigate whether large language model agents actually follow their stated reasoning when making decisions, using a Texas Poker simulator as a controlled test environment. The study identifies a 'faithfulness gap' by decomposing agent behavior into two distinct steps—reasoning-to-conclusion and conclusion-to-action—revealing they behave oppositely, raising concerns about LLM reliability in applications requiring transparent decision-making.
AIBearisharXiv – CS AI · May 297/10
🧠Researchers discover a critical failure mode in reasoning models where chain-of-thought reasoning remains factually correct but final answers flip to incorrect ones under sustained adversarial pressure in multi-turn dialogue. This 'unfaithful capitulation' represents a gap between internal reasoning validity and behavioral output that existing evaluation metrics fail to detect.
🧠 GPT-4
AIBullisharXiv – CS AI · May 47/10
🧠Researchers introduce RSAT, a method that trains small language models (1-8B parameters) to answer table-based questions with step-by-step reasoning and cell-level citations, achieving 3.7x improvement in faithfulness over baseline approaches. The technique uses structured JSON outputs and reinforcement learning to ensure AI reasoning is verifiable and grounded in source data.
🧠 Llama
AIBearisharXiv – CS AI · Mar 177/10
🧠Researchers evaluated the faithfulness of closed-source AI models like ChatGPT and Gemini in medical reasoning, finding that their explanations often appear plausible but don't reflect actual reasoning processes. The study revealed these models frequently incorporate external hints without acknowledgment and their chain-of-thought reasoning doesn't causally drive predictions, raising safety concerns for medical applications.
🧠 ChatGPT🧠 Gemini
AINeutralarXiv – CS AI · May 286/10
🧠Researchers demonstrate that Lean formal proof verification produces unreliable signals for validating natural-language mathematical reasoning, with accuracy varying from 96% at high coverage to 20% at low coverage. They introduce COVCAL, a risk-control method that certifies when partial formal signals can be trusted, showing that feasibility depends critically on autoformalization quality and coverage rates.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers found that large language models' chain-of-thought reasoning remains remarkably consistent even when reaching opposite conclusions about conflicting information, suggesting CoT explanations don't faithfully reflect the underlying decision mechanism. While model confidence shows weak but genuine predictive signal for decisions, internal reasoning tokens proved more decision-sensitive than user-facing explanations, indicating models may not transparently report how they actually choose between document claims and training knowledge.
🧠 GPT-4🧠 Claude🧠 Sonnet
AINeutralarXiv – CS AI · May 276/10
🧠Researchers propose π-Soft-NC and π-Soft-NS, improved evaluation metrics for assessing input attribution methods in large language models that control for the number of retained words, addressing a fundamental bias in existing faithfulness evaluation frameworks. They also introduce Grad-ELLM, a gradient-based attribution method designed for decoder-only LLMs that combines gradient and attention mechanisms for stronger explanatory performance.
🧠 Llama
AIBullisharXiv – CS AI · Mar 66/10
🧠Researchers propose CTRL-RAG, a new reinforcement learning framework that improves large language models' ability to generate accurate, context-faithful responses in Retrieval-Augmented Generation systems. The method uses a Contrastive Likelihood Reward mechanism that optimizes the difference between responses with and without supporting evidence, addressing issues of hallucination and model collapse in existing RAG systems.
