AIBullisharXiv – CS AI · 3d ago7/10
🧠Researchers present a unified evaluation framework for assessing LLM agentic capabilities, integrating 7 benchmarks across 24 domains with standardized testing methodology. The framework disentangles intrinsic model performance from implementation artifacts, revealing that scaffold choices and environmental volatility significantly impact benchmark results across 15 models tested.
🏢 Meta🏢 Hugging Face
AIBearisharXiv – CS AI · 3d ago7/10
🧠A comprehensive survey reveals that machine learning systems deployed in regulated financial sectors—credit risk, fraud detection, and anti-money laundering—suffer from reproducibility failures caused by hardware-level nondeterminism in neural networks and generative AI. The research quantifies specific vulnerabilities across tabular models, graph networks, and LLM-based workflows, proposing evaluation frameworks to improve auditability in financial AI systems.
AINeutralarXiv – CS AI · May 127/10
🧠A position paper proposes that NeurIPS implement mandatory reproducibility standards for frontier AI safety claims, arguing that the field's most consequential assertions about model safety are routinely made without releasing the artifacts needed to verify them. The proposal introduces a three-tier disclosure framework with controlled review mechanisms to address an evidential inversion where critical safety claims lack the rigor applied to less impactful research.
AIBullisharXiv – CS AI · May 97/10
🧠Researchers introduce execution lineage, a DAG-based execution model that makes AI-native workflows reproducible and maintainable by explicitly tracking dependencies and enabling identity-based replay. Tested against traditional loop-based approaches, the system demonstrated superior performance in preserving work integrity during updates while preventing unrelated context contamination.
AIBearisharXiv – CS AI · May 97/10
🧠A comprehensive study reveals that while AI adoption in research has surged exponentially since 2015, the technology remains concentrated in narrow domains tied to computer science with limited epistemological transformation. The research identifies concerning patterns including higher retraction rates in AI-supported work, citation inflation, and geographic disparities in adoption across countries and disciplines.
AIBearisharXiv – CS AI · May 77/10
🧠A comprehensive bibliometric audit reveals that academic papers evaluating large language models systematically lag behind frontier AI capabilities by a median of 10.85 points on the Epoch AI Capabilities Index, with this gap widening at 5.53 points annually. The study finds that most papers fail to disclose critical configuration details and make broad claims about "AI" capabilities rather than specific tested models, distorting how AI progress is understood in policy and media.
🧠 GPT-4🧠 GPT-5🧠 Claude
AIBearisharXiv – CS AI · May 47/10
🧠Researchers introduced AutoMat, a benchmark testing whether AI coding agents can reproduce computational materials science findings from academic papers. Current LLM-based agents achieved only 54.1% success rates, revealing significant limitations in reconstructing complex scientific workflows, interpreting domain-specific procedures, and validating results against original claims.
AIBearisharXiv – CS AI · Apr 107/10
🧠Researchers discovered that GPT-4o exhibits significant daily and weekly performance fluctuations when solving identical tasks under fixed conditions, with periodic variability accounting for approximately 20% of total variance. This finding fundamentally challenges the widespread assumption that LLM performance is time-invariant and raises critical concerns about the reliability and reproducibility of research utilizing large language models.
🧠 GPT-4
AINeutralarXiv – CS AI · Mar 177/10
🧠Researchers convened a February 2025 workshop to explore how meta-research methodologies can enhance Trustworthy AI (TAI) implementation in healthcare. The study identifies key challenges including robustness, reproducibility, clinical integration, and transparency gaps, proposing a roadmap for interdisciplinary collaboration between TAI and meta-research fields.
AINeutralarXiv – CS AI · Mar 177/10
🧠Researchers analyzed 3,550 papers to map the divide between AI Safety (AIS) and AI Ethics (AIE) communities, proposing a 'critical bridging' approach to reconcile tensions. The study identifies four engagement modes and finds overlapping concerns around transparency, reproducibility, and governance despite fundamental differences in approach.
AINeutralarXiv – CS AI · Mar 57/10
🧠A study reveals that 74% of healthcare AI research papers still use private datasets or don't share code, creating reproducibility issues that undermine trust in medical AI applications. Papers that embrace open practices by sharing both public datasets and code receive 110% more citations on average, demonstrating clear benefits for scientific impact.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers introduce MACC (Multi-Agent Collaborative Competition), a new institutional architecture that combines multiple AI agents based on large language models to improve scientific discovery. The system addresses limitations of single-agent approaches by incorporating incentive mechanisms, shared workspaces, and institutional design principles to enhance transparency, reproducibility, and exploration efficiency in scientific research.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers propose a standardized measurement protocol for evaluating retrieval-augmented generation (RAG) systems using LLM judges, addressing inconsistencies in how semantic search quality is assessed. The standard fixes key variables like evidence budget and prompt while requiring cluster-aware statistical testing, revealing that previous comparisons may have overstated progress and that traditional BM25 retrieval outperforms pure semantic methods under controlled conditions.
AINeutralarXiv – CS AI · 3d ago6/10
🧠ResearchLoop is a new technical framework that addresses reproducibility and auditability challenges in AI-assisted research by implementing an evidence-gated control plane. The system treats research components—questions, contracts, evidence, claims, and papers—as durable state objects, enabling verification of research claims throughout the AI-assisted workflow. The framework was validated through nine experimental versions, including self-hosting and mathematical olympiad benchmarks.
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduce Picid, a standardized evaluation infrastructure for Prognostics and Health Management (PHM) that addresses the reproducibility crisis in predictive maintenance across industries. The framework formalizes dataset construction, preprocessing, and evaluation metrics to enable fair comparisons of fault detection, diagnostics, and prognostics models across diverse domains like batteries, bearings, and engines.
🏢 Meta
AINeutralarXiv – CS AI · 3d ago6/10
🧠Researchers introduce an agentic, framework-based approach to reproducibly translate machine learning papers—specifically in Prognostics and Health Management (PHM)—into executable, comparable benchmark implementations. By mapping papers onto a shared framework with structured slot-binding interfaces, the method addresses critical reproducibility gaps caused by incomplete documentation, implicit design choices, and restricted dataset access.
AINeutralarXiv – CS AI · 3d ago6/10
🧠A reproducibility study of the TRIANGLE framework reveals that geometric alignment on hyperspheres improves multimodal retrieval beyond traditional pairwise approaches, achieving up to 8.7 point gains in zero-shot settings. However, researchers identified critical optimization instabilities when jointly training with data-text matching loss and reduced cross-dataset generalization with fine-tuning, suggesting the method's benefits are context-dependent rather than universally applicable.
AINeutralarXiv – CS AI · 4d ago6/10
🧠Researchers propose a unified evaluation framework for LLM-based agents, arguing that current benchmarks suffer from inconsistent methodologies, proprietary configurations, and environmental variability that obscure actual model performance. The lack of standardization hampers fair comparison and reproducibility across agent development, necessitating industry-wide evaluation standards.
AINeutralarXiv – CS AI · 4d ago6/10
🧠A comprehensive systematic review of 139 studies reveals that multimodal information fusion improves document classification accuracy by 5.28 percentage points, while multiview approaches provide modest gains of 4.67%. The research identifies critical gaps in methodological rigor, with less than 24% of studies employing statistical validation, highlighting the need for more robust research standards in the field.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers demonstrate that unpredictability in language agents does not equate to effective control, finding that structured decision-making mechanisms significantly outperform stochastic sampling across 74,352 test cases. The study challenges assumptions about randomness and control in AI systems, with implications for agent reliability and interpretability.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers demonstrate that EEG-based deep learning models produce unstable predictions when preprocessing pipelines change, with up to 42% of predictions flipping across different preprocessing choices. The study introduces three tools—Walsh-Hadamard decomposition, Preprocessing Uncertainty metrics, and a regularization approach—to measure and mitigate this instability, revealing a critical reliability gap in brain-computer interface systems.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers introduce Mage, a multi-axis evaluation framework that reveals compile-pass rate is a misleading metric for assessing LLM-generated code in complex domains. Testing across four open-weight language models on game scene synthesis, they find direct code generation achieves 43% runtime success but produces structurally invalid outputs, while IR-conditioned approaches recover functional correctness at the cost of lower raw execution rates.
AINeutralarXiv – CS AI · May 116/10
🧠Researchers introduce CyBiasBench, a benchmark revealing that LLM agents deployed for cybersecurity attacks exhibit inherent biases toward specific attack families regardless of prompting. The study demonstrates agents resist steering away from their preferred attack patterns, suggesting these biases are fundamental agent characteristics rather than prompt-dependent behaviors.
AIBearisharXiv – CS AI · May 16/10
🧠A comprehensive study comparing 12 large language models against 4 classical classifiers for automating evidence screening in software engineering systematic literature reviews reveals that LLMs exhibit significant performance variability and lack consistent superiority over traditional methods. The research emphasizes that abstract availability is critical for LLM performance, while title and keywords provide minimal additional value, suggesting LLM adoption should be driven by operational constraints rather than performance guarantees.
🏢 OpenAI🏢 Anthropic🧠 Gemini
AIBearisharXiv – CS AI · Apr 206/10
🧠A new study reveals that using large language models to generate synthetic datasets ("silicon samples") produces highly variable results depending on configuration choices, with correlation outcomes ranging from r=.23 to r=.84 on the same task. This demonstrates that analytic flexibility in LLM-based data generation poses a significant threat to research validity and reproducibility in social science applications.
