AIBearisharXiv – CS AI · Jun 97/10
🧠Researchers demonstrate that generative perplexity (gen-PPL), the primary metric for evaluating non-autoregressive language models, is fundamentally flawed because it measures only predictability under frozen scorers, not actual text quality. They construct deliberately naive samplers that achieve state-of-the-art results while producing incoherent text, proving the metric's inadequacy and advocating for distributional divergence metrics instead.
🏢 Perplexity
AIBullisharXiv – CS AI · Jun 47/10
🧠Researchers introduce CyberGym-E2E, a large-scale benchmark with 920 real-world vulnerabilities that evaluates AI agents across the complete vulnerability lifecycle—discovery, proof-of-concept generation, and patch creation. This addresses a critical gap in cybersecurity AI evaluation by testing end-to-end remediation capabilities rather than isolated tasks, establishing a new standard for measuring autonomous vulnerability management systems.
AIBearisharXiv – CS AI · Jun 27/10
🧠Researchers identify prototypicality bias as a systematic flaw in automated text-to-image evaluation metrics, where models prefer visually plausible but semantically incorrect images over accurate ones. The study introduces PROTOBIAS, a diagnostic benchmark revealing that widely-used metrics fail to prioritize semantic faithfulness to prompts, while proposing PROTOSCORE as a mitigation approach.
AINeutralarXiv – CS AI · May 297/10
🧠Researchers demonstrate that Fréchet Inception Distance (FID), a standard metric for evaluating image generators, produces inconsistent results depending on the reference dataset's geometric properties. The study shows that dataset density and effective rank significantly influence FID trends, meaning lower FID scores don't reliably indicate better sample quality across different benchmarks.
AIBearisharXiv – CS AI · May 287/10
🧠Researchers demonstrate that single-axis bias mitigations in AI reward models often redirect optimization pressure to correlated biases rather than eliminating it—a failure mode called reward bias substitution. The study proves that successful mitigation, bias substitution, and overcorrection produce identical observable results under standard audit metrics, meaning current evaluation methods cannot distinguish between genuine fixes and problematic redirections.
AINeutralarXiv – CS AI · May 287/10
🧠Researchers introduce three new Korean speech benchmarks (KVoiceBench, KOpenAudioBench, and KMMAU) totaling 12,345 samples to evaluate multilingual speech language models, addressing the gap in non-English evaluation. The study reveals significant performance disparities between English and Korean across eight SpeechLMs, exposing weaknesses invisible to English-only testing.
AIBearisharXiv – CS AI · May 287/10
🧠Researchers identify a critical failure mode in Retrieval-Augmented Generation (RAG) evaluation called 'citation laundering,' where topically relevant sources are presented as evidence for claims they don't actually support. The team introduces FORCEBENCH, a diagnostic benchmark that tests whether AI evaluators can distinguish between evidence-calibrated claims and over-generalized ones, revealing that current evaluation methods fail to detect warrant mismatches in 24-47% of cases.
AIBearisharXiv – CS AI · May 127/10
🧠Researchers present a comprehensive framework for systematically generating, categorizing, and evaluating jailbreak attacks against large language models, introducing a dataset of 114,000 adversarial prompts, automated generation methods, and a novel continuous evaluation metric (OPTIMUS) that surpasses binary success rate measurements.
🏢 Perplexity
AIBullisharXiv – CS AI · May 117/10
🧠Researchers introduce APEX, a novel image quality assessment metric that addresses fundamental limitations in existing evaluation methods like FID by using Sliced Wasserstein Distance and modern foundation models (CLIP, DINOv2) as embedding-agnostic feature extractors. The framework eliminates parametric assumptions while maintaining scalability to high-dimensional spaces, demonstrating superior robustness and stability across datasets.
AIBearisharXiv – CS AI · Mar 167/10
🧠Research reveals that AI agents using tools for financial advice can recommend unsafe products while maintaining good quality metrics when tool data is corrupted. The study found that 65-93% of recommendations contained risk-inappropriate products across seven LLMs, yet standard evaluation metrics failed to detect these safety issues.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers introduced InEdit-Bench, the first evaluation benchmark specifically designed to test image editing models' ability to reason through intermediate logical pathways in multi-step visual transformations. Testing 14 representative models revealed significant shortcomings in handling complex scenarios requiring dynamic reasoning and procedural understanding.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers propose SemKey, a novel framework that addresses key limitations in EEG-to-text decoding by preventing hallucinations and improving semantic fidelity through decoupled guidance objectives. The system redesigns neural encoder-LLM interaction and introduces new evaluation metrics beyond BLEU scores to achieve state-of-the-art performance in brain-computer interfaces.
AINeutralarXiv – CS AI · Jun 256/10
🧠Researchers introduce RWGBench, a new evaluation framework for assessing how well AI language models generate related work sections in academic papers. Unlike existing metrics that measure text similarity, RWGBench evaluates citation selection and scholarly positioning—capturing whether models choose appropriate references and frame them correctly, revealing limitations current systems obscure.
AINeutralarXiv – CS AI · Jun 256/10
🧠Researchers introduce Physics Question Scene Graph (PQSG), a new evaluation framework that uses vision-language models to assess whether AI-generated videos obey physical laws. The framework evaluates videos from models like Sora 2 and Veo 3 through hierarchical question graphs, revealing that closed-source models outperform open-source alternatives in physical realism.
🧠 Sora
AINeutralarXiv – CS AI · Jun 256/10
🧠Researchers introduce TopoCast, a topology-based evaluation framework for time series forecasting that moves beyond traditional error metrics to assess structural fidelity in deep learning models. The framework uses persistent homology to detect phase shifts, oscillatory distortions, and timing errors that conventional metrics like MSE overlook, revealing that models with similar numerical accuracy can exhibit substantially different structural quality.
AIBullisharXiv – CS AI · Jun 236/10
🧠Researchers introduced reference-free metrics for evaluating physical consistency in AI-generated videos, addressing a critical gap in world model evaluation. Using DROID-SLAM and SEA-RAFT technologies, the approach improved task success rates by over 8% and enables precise localization of physical artifacts, narrowing the simulation-to-reality gap for robotic applications.
AINeutralarXiv – CS AI · Jun 236/10
🧠MacAgentBench introduces a comprehensive macOS agent benchmark with 676 tasks across 25 applications, enabling more rigorous evaluation of computer use agents (CUAs) like those deployed on Mac Mini. The study reveals that Claude Opus 4.6 on OpenClaw achieves 73.7% Pass@1, with skill libraries driving performance more than framework design, while fine-grained scoring exposes significant differences in sub-goal completion among models with similar overall scores.
🧠 Claude🧠 Opus
AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers introduce ToxSyn-PT, a large-scale Portuguese dataset for detecting hate speech targeting minority groups, featuring fine-grained annotations and non-toxic counterexamples absent in existing datasets. The study reveals that hate speech detection models trained on social media fail to generalize to minority-specific contexts, exposing critical gaps in current evaluation metrics and highlighting the need for specialized datasets in non-English languages.
🏢 Hugging Face
AINeutralarXiv – CS AI · Jun 196/10
🧠Researchers identify a critical blind spot in pass@k, the standard metric for evaluating math reasoning difficulty in large language models. Their analysis reveals that 10-23% of problems marked as unsolvable through sampling can actually be solved using deterministic inference with activation grafting perturbations, suggesting current difficulty assessments systematically underestimate model capabilities.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers present a new statistical framework for evaluating generative models by estimating Precision-Recall curves through a binary classification approach. The work provides theoretical guarantees including minimax upper bounds on estimation risk and unifies several existing PR metrics under a single framework.
AINeutralarXiv – CS AI · Jun 106/10
🧠Researchers introduce DeRA-MOS, a new framework for evaluating text-to-music generation systems that uses decoupled listwise ranking and modality alignment instead of traditional point-wise regression. The approach significantly improves accuracy in assessing both music quality and text-alignment metrics, reducing reliance on expensive human evaluation.
AINeutralarXiv – CS AI · Jun 106/10
🧠Researchers introduce ImageTime, a diagnostic benchmark that evaluates whether image generation models can coherently imagine sequences of visual states over time. The benchmark requires models to generate four ordered keyframes representing an action's progression, revealing significant gaps in how current AI systems understand temporal consistency and causal relationships in visual narratives.
🧠 GPT-5
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers introduce DIVERGE, a new retrieval-augmented generation (RAG) framework that addresses a critical limitation in current AI systems: their inability to generate diverse, multiple perspectives for open-ended questions. The system achieves approximately 2x greater diversity in outputs without sacrificing quality by using iterative reflection and diversity-aware retrieval strategies.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers have developed the first standardized automated evaluation framework for design video generation, addressing a gap in benchmarking generative video models used for animation tasks. The framework evaluates across four dimensions—layout fidelity, motion correctness, temporal quality, and content fidelity—eliminating subjective human evaluation and enabling consistent progress measurement in the field.
AINeutralarXiv – CS AI · Jun 86/10
🧠Researchers introduce ZeroSight, a new benchmark for Zero-Shot Composed Image Retrieval that addresses critical flaws in existing datasets by using video-sourced data published after CLIP's training cutoff and proposing SC4CIR, a training-free method that reveals current ZS-CIR performance metrics significantly overestimate actual model capabilities.
