AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers evaluated whether structural codebase indexing improves coding agent performance by running controlled experiments with Claude Opus 4.7 across standardized benchmarks. Results show the index significantly improves code localization and task resolution rates without increasing costs, and outperforms simpler retrieval baselines, suggesting structural ranking becomes valuable for multi-file code changes.
🧠 Claude🧠 Opus
AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers introduce AInterviewer, an open-source platform that combines large language models with traditional survey software to conduct automated qualitative interviews while maintaining data security and reproducibility. Unlike proprietary solutions, the system runs on locally hosted models and enforces standardized question administration, addressing concerns about privacy and scientific rigor in AI-driven research.
AINeutralarXiv – CS AI · Jun 236/10
🧠Researchers introduce ARVO, a large-scale dataset of over 6,100 reproducible vulnerabilities from open-source software projects, addressing a critical gap in security research by prioritizing reproducibility alongside scale and diversity. The dataset achieves 81% successful vulnerability reproduction and 89.4% patch identification accuracy, enabling automated analysis and direct vulnerability interaction capabilities absent in existing datasets.
AINeutralarXiv – CS AI · Jun 236/10
🧠DEMM-Bench introduces a benchmark framework for evaluating whether evidence records in agent-runtime systems sufficiently answer governance questions about specific decisions. Using the Decision Evidence Maturity Model, researchers tested 64 cases across eight evidence regimes and found that existing baselines overclaim sufficiency in 50-75% of cases, while a property-level scorer achieved 56.25% accuracy with zero overclaims.
AINeutralarXiv – CS AI · Jun 115/10
🧠SemantiClean is a modular framework that extracts semantic signals from e-commerce session data to predict purchase intent and customer behavior while prioritizing auditability and reproducibility over raw predictive accuracy. The system uses a predefined library of 24 behavioral elements organized across four layers and implements safeguards against signal inflation, representing a shift toward transparent, governance-focused AI systems over conventional black-box optimizers.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers propose extending preregistration practices from human subjects research to AI agent experiments, addressing methodological vulnerabilities introduced by the ease of iterating on model selection, prompts, and experimental settings. The paper catalogs researcher degrees of freedom that make p-hacking and selective reporting easier to exploit in AI experiments while remaining difficult to detect, and calls for journals and conferences to adopt standardized preregistration templates.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers present a staged-promotion protocol for efficiently screening machine learning configurations during micro-pretraining, using fixed budget increments across heterogeneous hardware to reduce experimental costs while mitigating the risk of selecting configurations that perform well only at tiny scales. The study demonstrates that early-stage rankings are unstable across hardware types, but a frozen promotion rule successfully identified a consistent top performer while reducing total GPU-hours from 432 to 169.2.
AINeutralarXiv – CS AI · Jun 116/10
🧠Researchers introduce DuoBench, a comprehensive benchmarking framework for evaluating bimanual robotic manipulation policies on the FR3 Duo platform. The framework includes eleven tasks implemented in simulation and real-world settings, with reproducible recipes and human-teleoperated datasets that reveal significant challenges in current dual-arm AI policies, particularly in coordination and sim-to-real transfer.
AIBearisharXiv – CS AI · Jun 106/10
🧠A complementary study of PlanGPT, an LLM-based automated planning system, challenges its effectiveness by re-evaluating its performance against traditional planners using metrics like plan cost and generation time. The research questions whether planning with large language models is truly beneficial, finding that PlanGPT performs no better than basic greedy search strategies.
AINeutralarXiv – CS AI · Jun 96/10
🧠A new arXiv paper analyzes the sources of variability in agentic AI systems, distinguishing between token-sampling randomness intrinsic to foundation models and external factors like environmental changes and infrastructure effects. The research clarifies when AI agent outputs are genuinely stochastic versus reproducible, with implications for understanding AI reliability in production deployments.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers present MedSci Skills, an open-source toolkit that pairs LLM-assisted clinical manuscript generation with deterministic verification gates to detect fabricated citations, numerical errors, and missing reporting guidelines. The system demonstrates 100% detection of seeded defects versus 41% for generic LLM reviewers, providing an auditable trail for biomedical publishing.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers propose an operational framework for evaluating recursive self-design in AI systems, where AI assists in modifying its own development mechanisms. The paper maps existing systems against four criteria and reports that Darwin Goedel Machine achieved significant performance improvements (20% to 50% on SWE-bench, 14.2% to 30.7% on Polyglot benchmarks) through iterative self-improvement over 80 cycles.
🏢 Meta
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers introduce Evaluation Cards, a standardized reporting framework that addresses fragmented AI evaluation practices across leaderboards and model cards. The system consolidates benchmark metadata, evaluation data, and model information into unified records with interpretive signals for reproducibility and comparability, deployed across 5,816 models and 635 benchmarks.
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers introduce a bidirectional search task linking code snippets with text descriptions and vice versa, addressing the gap between scientific publications and their implementations. They present a large dataset with automatically-generated training data and manually-annotated test sets, along with a modular encoder-based approach that achieves strong in-domain results with promising out-of-domain generalization.
🧠 GPT-4
AINeutralarXiv – CS AI · Jun 96/10
🧠Researchers present a rigorous study of fine-tuning OpenAI's Whisper model for Swiss German speech recognition, achieving 25.6% WER with honest evaluation on disjoint test data. The work exposes significant benchmark contamination in published Swiss German ASR results, revealing that previous state-of-the-art claims were inflated by models memorizing test sets rather than genuinely understanding dialect.
🏢 OpenAI🏢 Nvidia
AINeutralarXiv – CS AI · Jun 95/10
🧠MIRAGE is a metadata-enriched framework for analyzing Mining Software Repositories (MSR) datasets from 2013-2024, incorporating FAIRness assessments and topic modeling to improve dataset discoverability and reusability. The research demonstrates that repository hosting sites and data formats significantly influence citation patterns and dataset utility in software engineering research.
AINeutralarXiv – CS AI · Jun 56/10
🧠Researchers evaluated how large language models performing structured data extraction from clinical notes respond to variations in prompts, model sizes, and data schemas. The study found that schema design—particularly the distinction between absent versus undocumented information—drives disagreement more than prompt phrasing, while model choice significantly impacts multi-class categorization tasks.
AINeutralarXiv – CS AI · Jun 56/10
🧠Researchers propose an LLM-integrated interface for mortality forecasting that translates natural language inputs into structured actuarial predictions while maintaining statistical rigor. The system uses a constrained orchestration layer to enhance accessibility for non-expert users without compromising reproducibility or analytical validity in high-stakes forecasting workflows.
AINeutralarXiv – CS AI · Jun 46/10
🧠Researchers conducted a reproducibility study of Vul-RAG, a RAG-based framework for detecting software vulnerabilities using LLMs, and found that while results are reproducible with open-weight models, performance plateaus around 0.30 pairwise accuracy regardless of model sophistication. The findings suggest that simply scaling up model capacity does not substantially improve vulnerability detection capabilities.
AINeutralarXiv – CS AI · Jun 26/10
🧠A comprehensive audit of 1,603 NLP papers from 2018-2025 reveals that while researchers increasingly report operational annotation details like recruitment and expertise, critical information for assessing data validity—such as annotator training, language proficiency, compensation, and inter-annotator agreement—remains frequently omitted. The study establishes a scalable framework and reporting taxonomy to improve reproducibility and reliability in NLP research.
AIBullisharXiv – CS AI · Jun 26/10
🧠Researchers have released MGRegBench, the first large-scale public dataset for mammography image registration with over 5,000 image pairs and 100 manually annotated landmarks. This addresses a critical gap in medical AI research by enabling standardized, reproducible benchmarking of registration methods across classical, learning-based, and deep learning approaches.
🏢 Meta
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers demonstrate that batch size is a critical hyperparameter systematically overlooked in LoRA fine-tuning evaluations, causing conflicting performance claims across variants. A cost-efficient tuning strategy reveals batch size's substantial impact on optimal model performance, reconciling previous contradictory results and establishing clearer evaluation standards.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers introduced GAIATrace, a token-level trace dataset documenting how state-of-the-art agentic AI systems (MiroThinker and OWL) execute general tasks, alongside Vidur-Agent, a simulator enabling reproducible system evaluation. This work addresses the black-box nature of agentic AI by providing unprecedented visibility into reasoning processes and system-level behavior.
AINeutralarXiv – CS AI · Jun 26/10
🧠SUPREME is an open-source framework that accelerates machine unlearning evaluation by distributing computation across multiple GPUs, addressing a critical bottleneck in AI model evaluation. The framework enables reproducible testing of data removal methods at scale, which has implications for privacy-preserving AI development and regulatory compliance.
AINeutralarXiv – CS AI · Jun 26/10
🧠Agentic-J is a containerized AI assistant system designed for ImageJ/Fiji that enables biologists to perform complex microscopy image analysis tasks using natural language commands. The system generates executable, documented scripts with specialized sub-agents handling plugin management, code generation, debugging, and statistical reporting, making advanced image analysis more accessible to researchers without extensive programming expertise.
