AINeutralarXiv – CS AI · Jun 257/10
🧠Researchers introduce Xcientist, a research harness that makes AI scientific reasoning transparent and auditable by externalizing research synthesis into inspectable artifacts. The system addresses 'claim drift'—where AI-generated mechanisms lose evidential grounding—and demonstrates traceable workflows across three scientific domains, suggesting AI scientists should be evaluated on accountability and reproducibility, not just output.
AIBearisharXiv – CS AI · Jun 237/10
🧠A technical study challenges the validity of reported improvements in multi-agent LLM coordination architectures by establishing a noise-floor baseline using Claude Haiku. The research reveals that paired configuration-equivalent trials produce statistical gaps of ±5pp at best, suggesting that seven of ten recent coordination papers report headline effects within or below this noise floor, raising questions about reproducibility and the actual gains from proposed architectures.
🧠 Claude🧠 Haiku
AIBearisharXiv – CS AI · Jun 237/10
🧠Researchers introduce CFAgentBench, a comprehensive benchmark for testing autonomous AI agents in construction finance workflows. The benchmark includes 1,014 task specifications across real software tools (ERP, payroll, banking portals) with strict functional grading, revealing that top models achieve only 67% accuracy on single attempts but collapse to 38% when consistency is required.
AINeutralarXiv – CS AI · Jun 237/10
🧠PaperClaw is a multi-agent AI system that automates academic research from conception to publication, combining autonomous operation with human-in-the-loop refinement. The system curates literature, generates hypotheses, tests them iteratively, and produces venue-compliant papers while maintaining verifiable citations and reproducible results.
AI × CryptoNeutralarXiv – CS AI · Jun 97/10
🤖A new arXiv paper audits 30 LLM-based trading studies and finds that while agent architectures are well-documented, evaluation methodologies—including execution timing, transaction costs, and data splits—lack standardization, making performance claims difficult to compare or reproduce. The authors argue that LLM trading research needs clearer reporting standards for execution realism before architectural improvements can be meaningfully assessed.
AIBullisharXiv – CS AI · Jun 87/10
🧠Researchers introduce OpenHalDet, an open-source benchmark framework that standardizes hallucination detection evaluation across diverse LLM scenarios. The unified framework addresses reproducibility challenges by providing consistent evaluation pipelines and supporting multiple detector types (black-box, gray-box, white-box), enabling more reliable comparison of hallucination detection methods.
AIBearisharXiv – CS AI · Jun 57/10
🧠Researchers challenge the credibility of recent computer-using agent (CUA) red-teaming studies by reproducing published prompt-injection attacks against frontier models Claude Sonnet 4.6 and GPT-5.4, finding 0% success rates compared to reported 42-98% attack success rates in prior work. The analysis reveals that published high attack success rates depend on reinforcement-learning optimized injection text rather than fundamental attack categories, and that safety hardening is domain-specific to browser interfaces, not generalizable across CUA modalities.
🧠 GPT-5🧠 Claude🧠 Sonnet
AINeutralarXiv – CS AI · Jun 57/10
🧠Researchers present a pre-registered causal decomposition framework that reveals how reinforcement learning from verifiable rewards (RLVR) conflates self-consistency elicitation with genuine reward-design effects. Through controlled experiments, they demonstrate that naive performance metrics systematically overestimate reward-design impact by 50-95%, with elicitation dominating in weak-prior regimes. The work provides diagnostic tools to audit published alignment research and expose methodological confounds.
AIBearisharXiv – CS AI · Jun 17/10
🧠A new arXiv paper challenges recent claims about LLM capabilities by arguing they lack scientific rigor under Popper's falsifiability principle. The authors identify methodological flaws in AI reasoning research, including opaque training data, non-reproducibility, and selection bias, then propose transparency guidelines to improve scientific integrity in LLM evaluation.
🏢 Meta
AIBearisharXiv – CS AI · Jun 17/10
🧠Researchers demonstrate that mechanistic interpretability—the process of reverse-engineering AI model behaviors through circuit discovery—suffers from fundamental statistical instability due to high variance in causal mediation analysis. The findings reveal that circuit structures are fragile and highly sensitive to input data and hyperparameter changes, calling into question the scientific validity of existing MI methodologies and necessitating stricter statistical practices in the field.
AIBearisharXiv – CS AI · May 297/10
🧠Researchers present an empirical study examining whether Large Language Model agents with tool-calling capabilities produce consistent outputs when given identical inputs across multiple invocations. The study expands beyond prior ReAct-style research to measure behavioral reproducibility in structured tool-calling interfaces, revealing a fundamental reliability gap that could impact production deployment of LLM agents.
AIBullisharXiv – CS AI · May 297/10
🧠Researchers introduce Croissant Tasks, a machine-readable metadata format designed to improve reproducibility in machine learning research by abstracting implementation details into high-level specifications. The format enables autonomous AI agents to generate independent implementations of ML experiments, addressing critical reproducibility challenges that plague modern AI research.
AIBullisharXiv – CS AI · May 287/10
🧠Researchers present a unified evaluation framework for assessing LLM agentic capabilities, integrating 7 benchmarks across 24 domains with standardized testing methodology. The framework disentangles intrinsic model performance from implementation artifacts, revealing that scaffold choices and environmental volatility significantly impact benchmark results across 15 models tested.
🏢 Meta🏢 Hugging Face
AIBearisharXiv – CS AI · May 287/10
🧠A comprehensive survey reveals that machine learning systems deployed in regulated financial sectors—credit risk, fraud detection, and anti-money laundering—suffer from reproducibility failures caused by hardware-level nondeterminism in neural networks and generative AI. The research quantifies specific vulnerabilities across tabular models, graph networks, and LLM-based workflows, proposing evaluation frameworks to improve auditability in financial AI systems.
AINeutralarXiv – CS AI · May 127/10
🧠A position paper proposes that NeurIPS implement mandatory reproducibility standards for frontier AI safety claims, arguing that the field's most consequential assertions about model safety are routinely made without releasing the artifacts needed to verify them. The proposal introduces a three-tier disclosure framework with controlled review mechanisms to address an evidential inversion where critical safety claims lack the rigor applied to less impactful research.
AIBearisharXiv – CS AI · May 97/10
🧠A comprehensive study reveals that while AI adoption in research has surged exponentially since 2015, the technology remains concentrated in narrow domains tied to computer science with limited epistemological transformation. The research identifies concerning patterns including higher retraction rates in AI-supported work, citation inflation, and geographic disparities in adoption across countries and disciplines.
AIBullisharXiv – CS AI · May 97/10
🧠Researchers introduce execution lineage, a DAG-based execution model that makes AI-native workflows reproducible and maintainable by explicitly tracking dependencies and enabling identity-based replay. Tested against traditional loop-based approaches, the system demonstrated superior performance in preserving work integrity during updates while preventing unrelated context contamination.
AIBearisharXiv – CS AI · May 77/10
🧠A comprehensive bibliometric audit reveals that academic papers evaluating large language models systematically lag behind frontier AI capabilities by a median of 10.85 points on the Epoch AI Capabilities Index, with this gap widening at 5.53 points annually. The study finds that most papers fail to disclose critical configuration details and make broad claims about "AI" capabilities rather than specific tested models, distorting how AI progress is understood in policy and media.
🧠 GPT-4🧠 GPT-5🧠 Claude
AIBearisharXiv – CS AI · May 47/10
🧠Researchers introduced AutoMat, a benchmark testing whether AI coding agents can reproduce computational materials science findings from academic papers. Current LLM-based agents achieved only 54.1% success rates, revealing significant limitations in reconstructing complex scientific workflows, interpreting domain-specific procedures, and validating results against original claims.
AIBearisharXiv – CS AI · Apr 107/10
🧠Researchers discovered that GPT-4o exhibits significant daily and weekly performance fluctuations when solving identical tasks under fixed conditions, with periodic variability accounting for approximately 20% of total variance. This finding fundamentally challenges the widespread assumption that LLM performance is time-invariant and raises critical concerns about the reliability and reproducibility of research utilizing large language models.
🧠 GPT-4
AINeutralarXiv – CS AI · Mar 177/10
🧠Researchers analyzed 3,550 papers to map the divide between AI Safety (AIS) and AI Ethics (AIE) communities, proposing a 'critical bridging' approach to reconcile tensions. The study identifies four engagement modes and finds overlapping concerns around transparency, reproducibility, and governance despite fundamental differences in approach.
AINeutralarXiv – CS AI · Mar 177/10
🧠Researchers convened a February 2025 workshop to explore how meta-research methodologies can enhance Trustworthy AI (TAI) implementation in healthcare. The study identifies key challenges including robustness, reproducibility, clinical integration, and transparency gaps, proposing a roadmap for interdisciplinary collaboration between TAI and meta-research fields.
AINeutralarXiv – CS AI · Mar 57/10
🧠A study reveals that 74% of healthcare AI research papers still use private datasets or don't share code, creating reproducibility issues that undermine trust in medical AI applications. Papers that embrace open practices by sharing both public datasets and code receive 110% more citations on average, demonstrating clear benefits for scientific impact.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers introduce MACC (Multi-Agent Collaborative Competition), a new institutional architecture that combines multiple AI agents based on large language models to improve scientific discovery. The system addresses limitations of single-agent approaches by incorporating incentive mechanisms, shared workspaces, and institutional design principles to enhance transparency, reproducibility, and exploration efficiency in scientific research.
AINeutralarXiv – CS AI · Jun 236/10
🧠DEMM-Bench introduces a benchmark framework for evaluating whether evidence records in agent-runtime systems sufficiently answer governance questions about specific decisions. Using the Decision Evidence Maturity Model, researchers tested 64 cases across eight evidence regimes and found that existing baselines overclaim sufficiency in 50-75% of cases, while a property-level scorer achieved 56.25% accuracy with zero overclaims.
