AINeutralarXiv – CS AI · Jun 256/10
🧠Researchers introduce RWGBench, a new evaluation framework for assessing how well AI language models generate related work sections in academic papers. Unlike existing metrics that measure text similarity, RWGBench evaluates citation selection and scholarly positioning—capturing whether models choose appropriate references and frame them correctly, revealing limitations current systems obscure.
AINeutralDecrypt · Jun 236/10
🧠Researchers testing strategic AI reasoning in Civilization VI observed an AI empire escalate to nuclear weapons development after falling behind in a cultural victory condition, ultimately failing to prevent its loss. The benchmark reveals limitations in AI strategic planning and escalation management when facing competitive pressure.
AIBearisharXiv – CS AI · Jun 236/10
🧠Researchers built Trucey, an AI coaching system for workplace negotiations, but found that a static handbook outperformed the conversational AI on user empowerment and usability. The study reveals that conversational AI imposes linear execution models on tasks requiring recursive, non-sequential preparation, challenging core assumptions about AI-mediated coaching design.
AIBearisharXiv – CS AI · Jun 236/10
🧠Researchers introduce EHR-Complex, a large-scale benchmark with 52K tasks for evaluating AI clinical agents on real-world electronic health record analysis. Testing reveals significant limitations, with top models achieving only 62.3% accuracy and exposure of three dominant failure modes: SQL logic errors, medical code lookup failures, and semantic misunderstandings.
AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers introduced AD-Bench, a real-world benchmark for evaluating LLM agents in advertising analytics tasks using actual production platform data. The framework addresses the gap between idealized benchmarks and practical agent performance, revealing that state-of-the-art models like Claude-Opus-4.7 struggle significantly with complex, multi-step advertising analytics despite achieving 76.9% accuracy on simpler tasks.
🧠 Claude
AIBearisharXiv – CS AI · Jun 196/10
🧠Researchers introduce BIM-Edit, a benchmark that evaluates large language models on their ability to edit existing Building Information Models in IFC format based on natural language instructions. The benchmark reveals significant capability gaps, with the best-performing LLM achieving only 49.5% accuracy and none solving more than 3.4% of tasks, highlighting that current AI systems struggle with the semantic preservation and relational understanding required for professional engineering workflows.
AIBearisharXiv – CS AI · Jun 116/10
🧠Researchers developed MentisOculi, a benchmark suite to test whether frontier multimodal AI models can use visual reasoning and mental imagery to solve complex problems. Testing shows that visual strategies—from latent tokens to generated images—fail to improve performance, revealing that despite their theoretical appeal, current models cannot effectively leverage visual thoughts for reasoning.
AIBearisharXiv – CS AI · Jun 116/10
🧠Researchers empirically tested whether open-source LLM-based AI agents can replace traditional Static Application Security Testing (SAST) tools like Bandit. The study found that current general-purpose open-source models underperform specialized security tools, suggesting agentic AI is not yet ready for autonomous vulnerability detection in real-world conditions.
AIBearisharXiv – CS AI · Jun 106/10
🧠Researchers introduce RealMath-Eval, a benchmark revealing that state-of-the-art LLM judges fail to accurately evaluate authentic student mathematical reasoning, performing significantly worse on real exam responses (MSE ~2.96) than on synthetic LLM-generated solutions (MSE ~1.17). The study identifies an "Evaluation Gap" stemming from human errors occupying a more diverse semantic space than the predictable patterns found in synthetic errors.
AINeutralarXiv – CS AI · Jun 106/10
🧠Researchers demonstrate a critical limitation in machine learning predictors: while they succeed at identified quantities, they collapse on unidentified counterfactual couplings, failing to capture uncertainty in causal relationships. The team proposes a mathematical framework using positive semidefinite coupling kernels to represent and bound these cross-world dependencies that standard prediction cannot recover.
AINeutralarXiv – CS AI · Jun 106/10
🧠Researchers at arXiv analyzed how large language models introduce distinctive emotional signatures when translating literary works, finding that LLM translations preserve author's voice less effectively than human translations. Post-editing partially corrects these emotional distortions, but MT systems consistently exhibit model-specific emotional fingerprints that deviate from human translation norms.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers evaluate whether deep research agents (DRAs) can improve iteratively through feedback, finding that self-reflection yields negligible gains while single rounds of process-level feedback produce substantial improvements—but these gains don't compound over multiple turns due to regression on previously satisfied criteria.
AINeutralarXiv – CS AI · Jun 86/10
🧠Researchers evaluated current large language models' effectiveness at solving exploration-exploitation tradeoffs in decision-making tasks. The study found that while reasoning models show promise for exploitation tasks, they remain impractical due to cost and speed constraints, and all tested LLMs underperform simple linear regression—though LLMs do excel at exploring large action spaces with semantic structure.
AIBearishThe Verge – AI · Jun 56/10
🧠Quilty, an AI startup claiming to predict film success from scripts alone, has faced significant credibility challenges after its predictions proved dramatically wrong in high-profile cases, incorrectly forecasting a box office flop over an Oscar-winning blockbuster. The failure highlights the persistent limitations of AI in predicting complex creative and commercial outcomes despite access to extensive data.
AIBearisharXiv – CS AI · Jun 56/10
🧠Researchers demonstrate that Large Language Models exhibit systematic convergence bias when mutating programs, revisiting similar structural forms in 87% of cases despite stochastic variation. This reveals a fundamental tension in LLM-driven program evolution: while these models excel at semantics-aware transformations, they inherently constrain exploration toward restricted regions of program space, limiting their effectiveness for open-ended evolutionary search.
AINeutralarXiv – CS AI · Jun 56/10
🧠Researchers compared AI-generated clinical literature summaries from three LLMs (Claude Sonnet, GPT-4o, and Llama 3.1) against expert-written summaries in headache medicine, finding that human experts still produced superior syntheses despite growing AI capabilities. The study reveals that while experts struggle to distinguish AI from human summaries, specialized domain knowledge and nuanced clinical reasoning remain difficult for current LLMs to fully replicate.
🧠 GPT-4🧠 Llama
AINeutralarXiv – CS AI · Jun 56/10
🧠Researchers introduce SoCRATES, a new benchmark for evaluating how well large language models can mediate conflicts across diverse scenarios and cultural contexts. Testing eight frontier LLMs reveals that even top-performing mediators resolve only about one-third of disagreements, with significant performance variations based on cultural identity, emotional reactivity, and party composition.
AINeutralarXiv – CS AI · Jun 56/10
🧠A research study evaluates whether current AI models can independently identify errors in published economic theory papers. The analysis finds that while AI-human collaboration can enhance peer review, no AI model successfully detected genuine errors without substantial human guidance, indicating significant limitations in AI's ability to advance theoretical knowledge autonomously.
🧠 ChatGPT🧠 Claude🧠 Gemini
AINeutralarXiv – CS AI · Jun 46/10
🧠Researchers introduced CodegenBench, a benchmark suite evaluating large language models' ability to generate efficient code across diverse CPU architectures including x86_64, Sunway, and Kunpeng. The study reveals that while LLMs excel at generating optimized code for mainstream architectures, they significantly underperform on domain-specific platforms with limited public documentation, exposing critical gaps in cross-platform generalization.
AIBearisharXiv – CS AI · Jun 46/10
🧠Researchers have discovered that text-to-image (T2I) models struggle with reasoning fidelity despite rendering visually clear text. The study reveals that current AI systems frequently produce semantic errors, logical inconsistencies, and incorrect reasoning steps when expressing complex solutions through images, highlighting a critical gap between visual and text-based reasoning performance.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce TECCI, a new benchmark dataset for evaluating text-guided image editing models, containing 7,550 image-instruction pairs across challenging edit types. Human evaluations reveal that leading image editors achieve only 22% success rates, with models struggling most on spatial reasoning and creative edits while excelling at color adjustments.
🧠 Gemini
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers conducted the first systematic evaluation of large language models' ability to understand pragmatic meaning conveyed through non-verbal responses in dialogue. The study found that LLMs experience up to 60% accuracy drops when interpreting non-verbal cues compared to verbal communication, revealing significant limitations in their understanding of indirect human communication.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce MIDI, a multilingual idiom dataset covering 18 languages across resource tiers, revealing that state-of-the-art NLP models struggle significantly with idiomatic expressions—particularly in low-resource languages and when interpreting literal meanings. The findings expose fundamental gaps in how current AI systems handle contextual language nuance across different linguistic communities.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce LinuxFLBench, a fault localization benchmark for Linux kernel bugs, and demonstrate that current LLM agents struggle with this complex task, achieving only 41.6% accuracy. They propose LinuxFL+, an enhancement framework that improves accuracy by 7.2-11.2% across all tested agents, addressing a critical gap in software debugging automation.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduce LLM-WikiRace, a benchmark that tests large language models' planning and reasoning abilities by requiring them to navigate Wikipedia links from a source to target page. While frontier models like Gemini-3 achieve superhuman performance on easy tasks, success rates plummet to 23% on hard difficulty, revealing significant limitations in long-horizon planning and recovery from failures.
🧠 GPT-5🧠 Claude🧠 Opus
