AINeutralarXiv – CS AI · 3d ago7/10
🧠Researchers introduce PMIYC, an automated framework for evaluating how effectively LLMs can persuade others and how susceptible they are to persuasion. Testing across multiple models reveals significant performance variations—GPT-4o shows 50% greater resistance to misinformation persuasion than Llama-3.3-70B, while o1-mini emerges as both persuasive and resistant, providing critical data for AI safety and alignment development.
🧠 GPT-4🧠 Claude🧠 Llama
AINeutralarXiv – CS AI · Mar 56/10
🧠Researchers introduce LifeBench, a new AI benchmark that tests long-term memory systems by requiring integration of both declarative and non-declarative memory across extended timeframes. Current state-of-the-art memory systems achieve only 55.2% accuracy on this challenging benchmark, highlighting significant gaps in AI's ability to handle complex, multi-source memory tasks.
AINeutralarXiv – CS AI · Mar 37/104
🧠Researchers introduce Interaction2Code, the first benchmark for evaluating Multimodal Large Language Models' ability to generate interactive webpage code from prototypes. The study identifies four critical limitations in current MLLMs and proposes enhancement strategies to improve their performance on dynamic web interactions.
AI × CryptoBullishOpenAI News · Feb 187/108
🤖OpenAI and Paradigm have launched EVMbench, a new benchmark tool designed to evaluate AI agents' capabilities in detecting, patching, and exploiting high-severity vulnerabilities in smart contracts. This collaboration represents a significant step toward improving smart contract security through AI-powered analysis tools.
AIBullishBlockonomi · 2d ago6/10
🧠Alibaba's Fun-Realtime-TTS-Preview voice AI model ranked fifth on the Artificial Analysis Speech Arena leaderboard, outperforming systems from OpenAI and xAI. This achievement marks Alibaba as the only Chinese-engineered voice system in the global top five, supporting 30+ languages and multiple Chinese dialects.
🏢 OpenAI🏢 xAI
AINeutralarXiv – CS AI · 2d ago6/10
🧠Researchers introduced OmniMatBench, a comprehensive multimodal reasoning benchmark containing 3,171 expert-curated problems across 19 materials science subfields. Evaluation of 13 major language models revealed significant gaps in AI reasoning capabilities, with the best model achieving only 37.2% accuracy, highlighting the need for improved scientific AI systems.
AINeutralarXiv – CS AI · 2d ago6/10
🧠Researchers introduce MPDocBench-Parse, a new benchmark dataset for evaluating multi-page document parsing systems across realistic, complex scenarios. The benchmark comprises 433 manually annotated documents spanning 3,246 pages in 15 document types, revealing that existing AI models excel at basic text extraction but struggle with semantic continuity, visual content preservation, and hierarchical structure recovery.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers have released IPO-Toolkit and IPO-Dataset, a comprehensive open-source framework and dataset containing over 109,000 IPO filings from 1994-2026 with 76,000+ extracted images. The resource enables large-scale analysis of long, multimodal financial documents and reveals that state-of-the-art AI models often misalign with expert judgments on financial chart interpretation tasks.
AINeutralarXiv – CS AI · May 76/10
🧠Researchers demonstrate a coding-agent system for ARC-AGI-3 that uses executable Python world models to solve abstract reasoning challenges without game-specific code. The agent achieved full solutions on 7 of 25 public games, establishing a generalizable baseline approach that relies on model verification and simplicity-driven refactoring rather than hand-coded logic.
AINeutralarXiv – CS AI · Apr 66/10
🧠Researchers introduced GBQA, a new benchmark with 30 games and 124 verified bugs to test whether large language models can autonomously discover software bugs. The best-performing model, Claude-4.6-Opus, only identified 48.39% of bugs, highlighting the significant challenges in autonomous bug detection.
🧠 Claude
AIBearishDecrypt · Mar 106/10
🧠BullshitBench, a new benchmark test, evaluates AI models' ability to detect nonsensical questions versus confidently providing incorrect answers. The results show most AI models fail this test, highlighting a significant reliability issue in current AI systems.
AINeutralarXiv – CS AI · Mar 66/10
🧠Researchers introduced FinRetrieval, a benchmark testing AI agents' ability to retrieve financial data, evaluating 14 configurations across major providers. The study found that tool availability dramatically impacts performance, with Claude Opus achieving 90.8% accuracy using structured APIs versus only 19.8% with web search alone.
🏢 OpenAI🏢 Anthropic🧠 Claude
AINeutralarXiv – CS AI · Mar 36/1011
🧠Researchers introduce LifeEval, a new multimodal benchmark designed to evaluate how well AI assistants can help humans in real-time daily life tasks from a first-person perspective. The benchmark reveals significant challenges for current AI models in providing timely and adaptive assistance in dynamic environments.
AINeutralarXiv – CS AI · Mar 37/108
🧠Researchers introduce PhotoBench, the first benchmark for personalized photo retrieval using authentic personal albums rather than web images. The study reveals critical limitations in current AI systems, including modality gaps in unified embedding models and poor tool orchestration in agentic systems.
AIBearisharXiv – CS AI · Mar 26/1018
🧠Researchers introduce FRIEDA, a new benchmark for testing cartographic reasoning in large vision-language models, revealing significant limitations. The best AI models achieve only 37-38% accuracy compared to 84.87% human performance on complex map interpretation tasks requiring multi-step spatial reasoning.
AIBullishOpenAI News · Dec 166/106
🧠OpenAI has launched FrontierScience, a new benchmark designed to test AI systems' reasoning capabilities across physics, chemistry, and biology. The benchmark aims to measure AI progress toward conducting actual scientific research tasks.
AIBullishOpenAI News · Nov 36/105
🧠OpenAI has launched IndQA, a new benchmark designed to evaluate AI systems' performance in Indian languages and cultural contexts. The benchmark covers 12 languages and 10 knowledge areas, developed in collaboration with domain experts to test cultural understanding and reasoning capabilities.
AINeutralOpenAI News · Apr 26/107
🧠PaperBench is a new benchmark designed to evaluate AI agents' ability to replicate state-of-the-art AI research. This tool aims to measure how effectively AI systems can reproduce complex research methodologies and findings.
AINeutralOpenAI News · Feb 186/106
🧠A new benchmark called SWE-Lancer has been introduced to evaluate whether frontier large language models can earn $1 million through real-world freelance software engineering work. This benchmark tests AI capabilities in practical, revenue-generating programming tasks rather than traditional academic assessments.
AINeutralarXiv – CS AI · Mar 35/104
🧠Researchers have introduced the TACIT Benchmark, a new programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains for evaluating AI models. The benchmark offers both generative and discriminative evaluation tracks with 6,000 puzzles and 108,000 images, using deterministic verification rather than subjective scoring methods.
$NEAR
AINeutralHugging Face Blog · Feb 45/106
🧠DABStep introduces a new benchmark for evaluating data agents' multi-step reasoning capabilities. The benchmark aims to assess how well AI agents can perform complex, sequential data analysis tasks that require multiple reasoning steps.
AINeutralHugging Face Blog · Mar 55/107
🧠ConTextual is a new benchmark or evaluation framework designed to test multimodal AI models' ability to jointly reason over both text and images in text-rich visual environments. This appears to be a research initiative focused on advancing AI capabilities in understanding complex visual-textual content.
AINeutralOpenAI News · Apr 104/106
🧠The article appears to discuss a new benchmark for measuring generalization capabilities in reinforcement learning (RL) systems. However, the article body was not provided, limiting the ability to analyze specific details about this RL benchmark.
