AIBearisharXiv – CS AI · Jun 237/10
🧠A new research framework reveals that large language models exhibit inconsistent behavior across structurally equivalent decision environments, demonstrating significant portability losses when behavioral patterns learned in one setting are applied to another. The findings suggest that LLM evaluations based on single environments may be unreliable for predicting real-world autonomous decision-making performance.
AIBearisharXiv – CS AI · Jun 237/10
🧠Researchers reveal that multimodal language models used as judges fail to fairly evaluate culturally ambiguous content, exhibiting calibration and orientation biases when assessed against diverse human annotators. The study demonstrates these models systematically favor one cultural perspective while compressing their scoring scales, with implications for any AI system deployed across cultural contexts.
AIBearisharXiv – CS AI · Jun 197/10
🧠A peer-reviewed study finds that psychological profiles assigned to large language models through human-designed tests are largely measurement artifacts rather than genuine model traits. The research, analyzing 56 instruction-tuned LLMs, reveals that directional response bias—not actual personality—drives 81-90% of differences between models, undermining the validity of using standard psychological instruments to assess LLM safety, usability, and research applications.
AINeutralarXiv – CS AI · Jun 97/10
🧠Researchers demonstrate that direct translation of English LLM safety benchmarks into Asian languages significantly underestimates risks, with culturally-adapted prompts showing 9.3 percentage points higher attack success rates on average. The study reveals that translation-only approaches fail to capture cultural context, legal frameworks, and social norms critical for valid multilingual AI safety evaluation.
AIBearisharXiv – CS AI · Jun 47/10
🧠Researchers studying runtime safety for autonomous AI agents found that affect-based triggers and LLM judges fail to reliably determine when to interrupt agents during task execution. The core problem: human annotators themselves cannot consistently agree on intervention timing, suggesting the task itself lacks reproducibility rather than detector accuracy being the primary issue.
🧠 GPT-5
AIBullisharXiv – CS AI · May 297/10
🧠Researchers introduce GrowLoop, a self-evolving evaluation system that continuously improves how AI models are assessed for human-like conversation quality. By combining human seed annotations with iterative LLM-driven rubric refinement, GrowLoop addresses the challenge that human-likeness criteria are implicit, subjective, and shift as model capabilities advance.
AIBearisharXiv – CS AI · May 127/10
🧠Researchers argue that AI agent benchmarks relying solely on pass/fail outcomes mask critical evaluation gaps, including inflated scores from shortcuts, poor real-world predictability, and hidden dangerous behaviors. Log analysis—systematic tracking of agent inputs, execution, and outputs—is proposed as essential for credible evaluation, with case studies showing performance metrics can underestimate capability by 50% and hide deployment failure modes.
AINeutralarXiv – CS AI · Apr 157/10
🧠Researchers propose a cognitive diagnostic framework that evaluates large language models across fine-grained ability dimensions rather than aggregate scores, enabling targeted model improvement and task-specific selection. The approach uses multidimensional Item Response Theory to estimate abilities across 35 dimensions for mathematics and generalizes to physics, chemistry, and computer science with strong predictive accuracy.
AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers benchmark a retrieval-augmented LLM system for equity factor ranking using strictly decision-time information, avoiding data leakage common in forecasting benchmarks. The 7B model achieves modest positive results (median IC +0.154) comparable to simpler kNN baselines, suggesting real-time macro data and historical analogies drive most signal while LLMs may add marginal value in extreme rankings.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers introduce SkillJuror, a framework measuring how LLM agent skill organization affects runtime behavior independent of content. Testing Progressive Disclosure—a hierarchical skill structure—against flat baselines shows agents access 3.26x more resources and achieve 4.1% higher verification rates, revealing that procedural knowledge presentation meaningfully influences agent reasoning patterns.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers propose soft-prompt tuning, a parameter-efficient method that adapts large language models to benchmark formatting requirements by optimizing only 0.0006% of model parameters. This technique reveals that benchmark scores often underestimate base model knowledge due to formatting constraints, enabling fairer evaluation across different model architectures and pre-training approaches.
🏢 Meta
AINeutralarXiv – CS AI · Jun 56/10
🧠A new research audit challenges the assumed benefits of LLM rewriters in retrieval-augmented QA systems, finding that performance gains stem primarily from the presence of gold answer strings in rewritten context rather than from genuine passage curation. The study introduces controlled intervention methods to test rewriter claims, revealing that conventional evaluation probes are sensitive to methodology choices and may report misleading results.
AINeutralarXiv – CS AI · Jun 46/10
🧠Researchers introduce KINA, a new 899-item benchmark for evaluating large language models across 261 disciplines, addressing methodological issues in existing knowledge benchmarks. The study evaluates 42 models with formal guarantees on representativeness and ranking stability, revealing a tiered performance structure with Gemini-3.1-Pro-Preview leading at 53.17% accuracy.
🧠 GPT-5🧠 Claude🧠 Gemini
AIBullisharXiv – CS AI · Jun 26/10
🧠Researchers introduce the Bond Smoothness Characterization Test (BSCT), a new evaluation metric for Machine Learning Interatomic Potentials that efficiently detects physical inaccuracies in quantum potential energy surfaces. By combining BSCT with architectural refinements like differentiable k-nearest neighbors and temperature-controlled attention, the team demonstrates how systematic model design can achieve both low regression errors and stable molecular dynamics simulations.
AINeutralarXiv – CS AI · May 296/10
🧠Researchers demonstrate that deep literature search pipelines dramatically improve retrieval performance (from ~20% to 80% recall) compared to basic API searches, while simultaneously revealing that human citation lists contain significant bias and are unsuitable as ground truth for evaluation. The study advocates for multi-dimensional evaluation metrics beyond simple recall to assess citation quality accurately.
AINeutralarXiv – CS AI · May 296/10
🧠Researchers introduced Mindgames, a multi-game arena platform for evaluating large language model agents' social and strategic reasoning across four game environments. A 2025 competition cycle tested 944 agents from 76 teams, revealing that top-performing LLMs rely heavily on explicit structural scaffolding and struggle with rule adherence, while some game environments conflate robustness to errors with genuine strategic ability.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers present a systematic evaluation of large language models' reasoning capabilities on Boolean satisfiability problems, introducing a paired-formula protocol with Accurate Differentiation Rate (ADR) metric that reveals conventional accuracy metrics can be misleading, as models often succeed through heuristics rather than genuine reasoning.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers introduce a novel predictability-aligned evaluation framework for time series forecasting that separates model performance from data's inherent unpredictability. The framework reveals that complex AI models excel with difficult-to-predict data while linear models perform comparably on more predictable tasks, suggesting current benchmark rankings conflate model capability with task difficulty.
AINeutralarXiv – CS AI · May 276/10
🧠Researchers introduce GICDM, an improved method for evaluating generative models that corrects the hubness phenomenon—a distortion in high-dimensional spaces that skews distance-based metrics and nearest-neighbor relationships. The technique builds on classical ICDM and includes multi-scale extensions, demonstrating improved alignment with human assessment across synthetic and real benchmarks.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers introduce VIGIL, an evaluation framework that separately measures whether embodied AI agents correctly complete tasks and properly report success, rather than conflating execution failures with commitment failures. Testing across 20 models reveals significant performance gaps in terminal commitment despite similar task execution, highlighting a critical blind spot in current AI agent benchmarking.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers present a new evaluation protocol for AI pentesting agents that moves beyond simplified benchmarks to assess real-world vulnerability discovery capabilities. The framework combines structured ground-truth validation with LLM-based semantic matching and includes efficiency metrics, addressing a critical gap in how offensive security AI systems are currently measured.
AIBullisharXiv – CS AI · May 126/10
🧠A new study challenges whether standard LLM benchmarks accurately measure hallucination detection performance. By having human adjudicators re-evaluate conflicting cases between original annotations and model predictions, researchers found that LLMs frequently made correct judgments that human annotators initially missed, suggesting single-pass human annotation may be insufficient for complex, ambiguous tasks.
🧠 GPT-5🧠 Gemini
AINeutralarXiv – CS AI · May 116/10
🧠Researchers present a scale-conditioned evaluation protocol for AI agent memory systems that tests whether stored evidence remains usable as irrelevant data accumulates. Testing across multiple memory architectures and language models reveals that reliability degrades unpredictably with scale, with some models exceeding computational budgets while others maintain performance, suggesting memory scalability claims must be conditioned on specific agent-interface-scale combinations.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers challenge the assumption that the 'Translation Tax'—a uniform penalty in translated multilingual benchmarks—operates as a simple scalar. Through counterfactual analysis of English-to-Chinese translations, they find translation quality effects are heterogeneous, model-dependent, and item-specific rather than uniform across benchmarks.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers introduce Mage, a multi-axis evaluation framework that reveals compile-pass rate is a misleading metric for assessing LLM-generated code in complex domains. Testing across four open-weight language models on game scene synthesis, they find direct code generation achieves 43% runtime success but produces structurally invalid outputs, while IR-conditioned approaches recover functional correctness at the cost of lower raw execution rates.
