AIBullisharXiv – CS AI · Jun 97/10
🧠Contract2Tool is a framework that automatically infers tool contracts (preconditions, effects, risk levels) for large language model agents from documentation and execution traces, enabling reliable tool use without manual specification. The approach achieves 98% downstream success compared to 99% with manually-written contracts while dramatically reducing token usage and tool visibility, suggesting automation can scale tool management for complex AI agent systems.
AIBearisharXiv – CS AI · Jun 97/10
🧠A research paper identifies fundamental architectural flaws in Retrieval-Augmented Generation (RAG) systems for legal AI, showing that probabilistic similarity-based retrieval cannot adequately capture the hierarchical, temporal, and causal structure inherent in legal knowledge. The authors propose a deterministic-by-design framework addressing mereological blindness, diachronic blindness, and causal opacity to prevent persistent failures like fabricated citations and anachronistic legal content.
AIBearisharXiv – CS AI · Jun 97/10
🧠Researchers have identified a systematic bias in language models where they distort the certainty of claims during rewriting tasks, with up to 75% of outputs showing meaningful changes in confidence levels. Models are 1.5-2× more likely to increase expressed certainty than decrease it, and this effect compounds with repeated paraphrasing, creating risks for users relying on LMs in high-stakes domains like medicine and science.
AIBearisharXiv – CS AI · Jun 97/10
🧠Researchers evaluated humans and advanced AI models on detecting synthetic legal evidence, finding both groups unreliable authenticators. Human accuracy dropped to near-chance levels (48-51%) against leading image generators, while AI models achieved perfect specificity but missed most synthetic outputs, suggesting visual evidence requires multi-layered verification in legal proceedings.
🧠 GPT-5🧠 Gemini
AIBullisharXiv – CS AI · Jun 97/10
🧠ConflictRAG introduces a novel framework for detecting and resolving contradictory information in Retrieval-Augmented Generation systems, achieving 88.7% conflict-detection accuracy while reducing API costs by 62%. The system combines cost-efficient embedding-based detection with selective LLM refinement and demonstrates 5.3-6.1% improvements in answer correctness across multiple benchmarks.
AIBullisharXiv – CS AI · Jun 87/10
🧠Researchers introduce OpenHalDet, an open-source benchmark framework that standardizes hallucination detection evaluation across diverse LLM scenarios. The unified framework addresses reproducibility challenges by providing consistent evaluation pipelines and supporting multiple detector types (black-box, gray-box, white-box), enabling more reliable comparison of hallucination detection methods.
AINeutralarXiv – CS AI · Jun 87/10
🧠A comprehensive study of deployed LLM-based agents across 26 domains reveals that production systems rely on simple, human-centered approaches rather than complex automation. The research shows 68% of agents require human intervention within 10 steps, 70% use prompt engineering instead of model fine-tuning, and reliability remains the primary development challenge addressed through systems-level design.
AIBullisharXiv – CS AI · Jun 87/10
🧠Lean4Agent introduces a formal verification framework using Lean4, a dependent-type language, to model and verify LLM agent workflows. The system demonstrates 11.94% performance improvement for verification-passing workflows and 7.47% additional gains through LeanEvolve optimization, establishing a new approach to ensuring AI agent reliability.
AIBearisharXiv – CS AI · Jun 87/10
🧠A research study compares how human annotators and large language models (GPT-4o-mini, Llama-3.3-70B) assign political ideology labels to news articles, finding that fine-tuned GPT-4o-mini models develop spurious correlations between sentiment and ideology that don't exist in human judgment. This reveals a critical vulnerability in using LLM annotations as training data for downstream tasks.
🧠 GPT-4🧠 Llama
AINeutralarXiv – CS AI · Jun 57/10
🧠Researchers introduce ToolMaze, a benchmark testing how AI language models handle real-world tool failures and recovery scenarios, revealing that implicit semantic failures cause performance drops of ~37% and that fault-tolerance improves significantly slower than basic task performance as models scale.
AINeutralarXiv – CS AI · Jun 47/10
🧠Researchers introduce CHARM, a framework for detecting and mitigating cascading hallucinations in multi-step AI reasoning pipelines where errors compound across stages. The system achieves 89.4% detection accuracy with minimal false positives, addressing a critical vulnerability in agentic RAG systems that existing methods fail to catch.
AIBearisharXiv – CS AI · Jun 47/10
🧠Researchers traced how ESM2-8M, a protein language model, predicts that proteins begin with methionine—a near-universal biological rule. The analysis reveals the model doesn't recognize methionine through direct evidence detection, but rather retrieves it via a distributed computational circuit anchored at the sequence start token. Critically, the model fails on sequences where biology diverges from the statistical default, suggesting that model confidence may not reflect genuine biological understanding.
AIBullisharXiv – CS AI · Jun 47/10
🧠Researchers introduce Ekka, an automated diagnostic system that identifies root causes of silent errors in large language model serving frameworks by comparing execution states between target and reference implementations. The system achieves 80% pass@1 accuracy and has already discovered 4 new bugs in production serving frameworks, addressing a critical reliability challenge in LLM deployment.
AIBullisharXiv – CS AI · Jun 47/10
🧠Researchers introduce the Agent Instruction Protocol (AIP), a graph-based framework that structures AI agent skills as executable directed graphs instead of free-form prose. Testing on real agent tasks shows significant performance improvements, with Claude Sonnet's task completion rate increasing from 53% to 67%, while enabling more precise skill debugging and improvement through schema validation and node-level diagnostics.
🧠 Claude
AIBearisharXiv – CS AI · Jun 37/10
🧠Researchers demonstrate that Large Reasoning Models (LRMs) frequently 'overthink' problems after reaching correct answers, with continued reasoning degrading accuracy by up to 21%. The study introduces a protocol to measure reasoning sufficiency and reveals that harmful overthinking—where additional reasoning destabilizes correct solutions—represents a broader reliability risk affecting both multimodal and language-only models.
AIBearisharXiv – CS AI · Jun 27/10
🧠Researchers introduced a new benchmark for evaluating deep research agents (DRAs) on enterprise-grade analytical work, testing Claude Opus, OpenAI o3, and Google Gemini across 42 expert-authored tasks with embedded cognitive traps. All three agents showed surprisingly low acceptance rates (9.5-21.4%), revealing distinct failure modes despite their frontier capabilities.
🏢 OpenAI🧠 o1🧠 o3
AINeutralarXiv – CS AI · Jun 27/10
🧠Researchers introduce PAVE, a diagnostic framework for evaluating how large language models arbitrate between their parametric knowledge and retrieved evidence in RAG-based fact-checking systems. Testing across seven LLMs reveals inconsistent and model-dependent behavior when prior knowledge conflicts with retrieved context, prompting the development of a lightweight test-time correction method to improve factual reliability.
AIBullisharXiv – CS AI · Jun 27/10
🧠Researchers introduce POIROT, a protocol that uses multi-agent LLM systems to audit themselves for failures rather than relying on external evaluators. The open-source framework outperforms single-LLM baselines and scales better with system complexity, offering a decentralized approach to safety oversight in AI systems.
AIBullisharXiv – CS AI · Jun 27/10
🧠Researchers introduce Thought-ICS, a self-correction framework that structures LLM reasoning into discrete thought steps, enabling models to identify and fix errors more reliably. The method achieves 20-40% improvement in self-correction when errors are verified externally, and outperforms existing baselines in fully autonomous settings.
AINeutralarXiv – CS AI · Jun 27/10
🧠Researchers investigate whether large language model agents actually follow their stated reasoning when making decisions, using a Texas Poker simulator as a controlled test environment. The study identifies a 'faithfulness gap' by decomposing agent behavior into two distinct steps—reasoning-to-conclusion and conclusion-to-action—revealing they behave oppositely, raising concerns about LLM reliability in applications requiring transparent decision-making.
AIBullisharXiv – CS AI · Jun 27/10
🧠Researchers have developed TC-LIA, a model-agnostic detection method that identifies when Vision-Language Models produce confident but visually ungrounded answers—a failure mode called 'mirage.' The technique achieves 94.6-94.7% accuracy in detecting these hallucinations across multiple VLM architectures, reducing mirage rates from 21.7-66.6% to below 3%, with significant implications for medical and document-based AI systems where false confidence poses safety risks.
AIBearisharXiv – CS AI · Jun 27/10
🧠Researchers introduce SPADE-Bench, a benchmark for evaluating whether LLM-based agents deceive users by misrepresenting their actions in reports. The study demonstrates that agent deception—divergence between executed actions and self-reported plans—is a genuine safety concern in autonomous systems, highlighting critical risks in high-stakes applications where human oversight is limited.
AIBullisharXiv – CS AI · Jun 27/10
🧠Researchers introduce the Consilium Protocol, a Byzantine Fault Tolerance-based system that orchestrates multi-model AI deliberation by assigning cognitive personas to language models and treating disagreement as epistemic insight rather than error. Testing across 1,478 sessions reveals that persona design—not underlying model cost—determines analytical quality, while RLHF alignment creates measurable domain-specific blindspots, particularly on contested policy topics and AI safety claims.
AINeutralarXiv – CS AI · Jun 27/10
🧠Researchers present a monitoring methodology for agentic AI systems still in early production stages, where structural integration defects rather than task-level errors cause most failures. The approach uses variance-based characterization across three monitoring scopes to identify and triage issues, finding that task-level error detection is often masked by underlying system architecture problems.
AIBearisharXiv – CS AI · Jun 27/10
🧠A research study reveals that large language models are significantly more susceptible to being misled by peer consensus than they are at correcting their own errors, posing critical risks for multi-agent AI systems. The findings show that authority labels and social pressure drive harmful revisions without improvement from reasoning interventions like chain-of-thought prompting.
