AIBearisharXiv – CS AI · Jun 257/10
🧠Researchers identify four specific failure modes in large language models attempting research-level mathematics: citation fabrication, premise smuggling, silent problem reformulation, and local-to-global compatibility gaps. Testing reveals that premise smuggling—where models assert unjustified claims as fundamental results—persists even when citations are accurate, suggesting retrieval-augmented generation alone cannot solve LLM reasoning failures.
🧠 Gemini
AIBearisharXiv – CS AI · Jun 257/10
🧠Researchers introduce TriViewBench, a controlled benchmark for evaluating multimodal AI models' ability to reason across multiple 3D views with varying complexity. Testing 18 MLLMs reveals a universal capability hierarchy and severe performance degradation on complex tasks, particularly in cross-view spatial reasoning, suggesting fundamental limitations in current AI architecture.
AIBearisharXiv – CS AI · Jun 237/10
🧠Researchers introduce RoboMME-Interference, a benchmark testing how robot memory systems perform across multiple sessions with irrelevant distractions. Testing current memory-augmented AI models reveals significant performance degradation as unrelated sessions accumulate, highlighting a critical gap in long-context robustness for real-world robot deployment.
AIBearisharXiv – CS AI · Jun 237/10
🧠Researchers introduce HOLMES, a new benchmark for evaluating higher-order logical reasoning in large language models, revealing that current LLMs struggle significantly with complex symbolic reasoning tasks that go beyond simple first-order logic. The benchmark demonstrates critical gaps in AI reliability, with the best-performing models achieving only 59.54% accuracy on tasks involving reasoning over rules, predicates, and constraints across legal and financial domains.
AIBearisharXiv – CS AI · Jun 197/10
🧠A new research framework called CWE-Trace challenges the claim that large language models can reliably detect software vulnerabilities, revealing that fine-tuned models achieve only 52.1% accuracy at best and lack genuine security reasoning despite appearing well-calibrated. The study of 834 Linux kernel samples shows that models exhibit systematic failure patterns that persist across datasets and resist correction through fine-tuning, suggesting they memorize patterns rather than understand vulnerability detection.
AIBearisharXiv – CS AI · Jun 117/10
🧠A comprehensive evaluation of frontier large language models for cybersecurity tasks reveals they struggle with high false positive rates (10-50%) in vulnerability detection and achieve only 4-8% accuracy in black-box testing, suggesting that specialized domain training and structured methodology matter more than model scale for security applications.
🧠 GPT-5🧠 Claude🧠 Gemini
AIBearisharXiv – CS AI · Jun 117/10
🧠Researchers introduced SciConBench, a benchmark evaluating AI agents' ability to synthesize scientific conclusions from systematic reviews. Testing eight frontier models and research agents under controlled conditions revealed fundamental limitations: the best-performing agent achieved only 0.337 factual F1 score, with consumer-facing tools like Google AI Overview generating incomplete or contradictory conclusions despite available ground-truth answers.
🏢 Google
AIBearisharXiv – CS AI · Jun 117/10
🧠Researchers demonstrate that Large Language Models systematically overestimate the novelty of AI-generated research questions compared to human expert assessment, revealing a critical gap in LLM-based scientific evaluation. The study introduces RQ-Bench, a benchmark showing that while LLMs rate model-generated questions as highly novel, domain experts prefer author-anchored reference questions and identify that many AI-generated questions lack depth or originality.
AI × CryptoBearishCrypto Briefing · Jun 117/10
🤖A recent study called 'Agents' Last Exam' reveals that AI agents successfully complete real-world work tasks only 2.6% of the time, exposing significant limitations in current AI model capabilities. This finding underscores the substantial gap between AI's theoretical potential and practical performance, necessitating major improvements in model architecture and training methodologies before widespread deployment in critical applications.
AIBearisharXiv – CS AI · Jun 107/10
🧠Researchers introduced PhysTool-Bench, a benchmark testing how well multimodal large language models (MLLMs) can recognize and use physical tools in real-world scenarios. Testing 13 leading models revealed significant limitations: even the best performer (Gemini-3.1-Pro) identified only 58.7% of tools in scenes and completed just 21% of end-to-end tasks, exposing critical gaps in perception and functional reasoning for embodied AI applications.
🧠 Gemini
AIBearishMIT News – AI · Jun 97/10
🧠A Media Lab study reveals that reliance on AI for news verification may paradoxically weaken users' ability to detect misinformation, similar to how GPS dependency has diminished navigation skills. This cognitive atrophy poses risks for media literacy and information security in an increasingly AI-mediated information ecosystem.
AIBearisharXiv – CS AI · Jun 97/10
🧠Researchers have identified a critical reliability flaw in multimodal large language models (MLLMs) used for video understanding: when the correct answer is absent from available options, these models fail to recognize it and instead select plausible incorrect alternatives. Testing across multiple models and benchmarks reveals this limitation is especially severe in temporal reasoning tasks and worsens with increased video frame sampling, with chain-of-thought prompting offering only partial mitigation.
AINeutralarXiv – CS AI · Jun 97/10
🧠Researchers introduced ResearchClawBench, a comprehensive benchmark with 40 tasks across 10 scientific domains designed to evaluate AI agents' ability to conduct autonomous scientific research. Current leading systems like Claude Code and Claude-Opus-4 score only 20-21.5 points, revealing significant gaps in experimental design, evidence synthesis, and scientific reasoning capabilities.
🧠 Claude
AINeutralarXiv – CS AI · Jun 97/10
🧠A comprehensive study challenges claims that large language models have surpassed human summarization capabilities, finding that while LLMs excel at surface-level coherence, human-written summaries remain superior in informativeness, faithfulness, and factuality—particularly for complex reasoning tasks.
AIBearisharXiv – CS AI · Jun 87/10
🧠Researchers reveal a significant gap between synthetic and real-world performance in LLM personalization systems by analyzing 550 human conversations across three stages: attribute extraction, attribute selection, and response generation. The study finds that current models struggle with human-aligned personalization and that learned reward models fail to adequately capture human preferences, highlighting fundamental limitations in how AI systems understand and incorporate user information.
AIBearisharXiv – CS AI · Jun 87/10
🧠A comprehensive study of 8 state-of-the-art language models reveals significant limitations in probabilistic reasoning, with accuracy dropping from 96% on standard problems to 59% on counterintuitive ones. The research demonstrates that LLMs are vulnerable to token bias and prompt manipulation, suggesting they lack genuine probability reasoning despite excelling at other mathematical tasks.
AIBearisharXiv – CS AI · Jun 57/10
🧠Researchers discovered that lexical density—the rate at which new information appears in text—significantly limits LLM effective context windows, causing near-perfect models to drop below 60% accuracy on information-dense contexts. This finding reveals that input length and needle position, traditionally blamed for context degradation, overlook a critical third factor that directly impacts real-world LLM performance on compact, information-rich data.
AINeutralarXiv – CS AI · Jun 57/10
🧠Researchers introduce CLASH, a dataset of 345 high-stakes dilemmas with 3,795 diverse perspectives, revealing that leading language models including GPT-4 and Claude struggle significantly with ambivalent value-based decisions. The study exposes fundamental limitations in LLM reasoning about conflicting values, with top models achieving only 24-51% accuracy on ambivalent scenarios, indicating a critical gap in AI systems designed for high-consequence decision-making.
🧠 GPT-5🧠 Claude
AIBearisharXiv – CS AI · Jun 27/10
🧠Researchers introduce ClinEnv, an interactive benchmark that evaluates large language models as attending physicians making real clinical decisions across multiple stages of patient care. The study reveals that even the strongest models achieve only 0.31 decision F1 scores, with significant gaps between diagnostic accuracy and clinical management quality, exposing how outcome-focused evaluations mask deficiencies in information-gathering processes.
AIBearisharXiv – CS AI · Jun 27/10
🧠Researchers introduce CardioLens, a rigorous evaluation framework revealing that state-of-the-art multimodal large language models (MLLMs) perform poorly at clinical cardiac MRI interpretation despite strong public benchmark results. The study demonstrates a significant gap between theoretical capabilities and real-world clinical applicability, with models failing to integrate distributed evidence across imaging sequences and temporal phases.
AIBearisharXiv – CS AI · Jun 27/10
🧠Researchers introduce Moment-Video, a benchmark revealing that current video multimodal large language models (MLLMs) struggle to understand brief, momentary visual events that last only a few frames. Testing 33 models shows the best achieves only 39.6% accuracy, exposing a critical gap in temporal fidelity that persists despite advances in general video understanding.
AIBearisharXiv – CS AI · Jun 27/10
🧠A new study reveals that large language models generate significantly less diverse arguments than humans when responding to public debates, with only 3.4% of LLM main arguments being unique compared to 65.3% for human responses. This 'argument collapse' phenomenon persists even when models are prompted to generate diverse answers, suggesting LLMs may homogenize public discourse by repeatedly introducing the same polished arguments across different contexts.
AIBearisharXiv – CS AI · Jun 17/10
🧠A new research paper reveals that self-organizing multi-agent LLM teams significantly underperform compared to their best individual expert members, with performance losses reaching 41.1% on ML benchmarks. The primary failure mechanism is not identifying experts but rather failing to leverage them appropriately, as teams tend toward consensus-averaging rather than expertise-weighted decision-making.
AIBearisharXiv – CS AI · Jun 17/10
🧠Researchers introduce LongDS, a benchmark revealing significant limitations in AI agents performing long-horizon data analysis tasks. Testing five state-of-the-art models shows best performance of only 48.45% accuracy with performance degrading by 47 points across task progression, indicating that maintaining analytical context over extended interactions remains a critical unsolved problem.
AIBearisharXiv – CS AI · May 297/10
🧠Researchers introduced FinVerBench, a benchmark for evaluating how well large language models verify financial statement accuracy using real SEC 10-K filings. Testing 14 contemporary LLMs revealed critical limitations: most models produced 95-100% false positives on clean statements, while performance varied dramatically based on how financial data was rendered, suggesting financial verification requires calibrated judgment beyond arithmetic detection.
🧠 Gemini
