AIBearisharXiv – CS AI · Jun 257/10
🧠Researchers demonstrate that machine unlearning methods that appear successful at the output layer—the standard evaluation metric—actually retain structured residual information in representation space compared to true retraining. This finding reveals a critical gap between apparent forgetting and genuine forgetting, suggesting current unlearning evaluations systematically overestimate effectiveness.
AINeutralarXiv – CS AI · Jun 127/10
🧠Researchers present DAF-AGI, a governance framework for defining artificial general intelligence, arguing that competing definitions of AGI produce contradictory verdicts on the same systems. The framework tests whether current generative AI systems qualify as AGI and finds certification only under performance-based metrics, while other approaches reject the claim, highlighting the necessity of definitional clarity before capability assessment.
AINeutralarXiv – CS AI · Jun 97/10
🧠A position paper argues that Anthropomorphic Misalignment Research (AMR) studies often lack sufficient empirical rigor to support critical AI safety decisions. The authors propose an evidence framework and diagnostic checklist to strengthen methodological standards and ensure AI risk claims rest on solid foundations.
AIBearisharXiv – CS AI · Jun 97/10
🧠Researchers conducted a large-scale analysis of human evaluation protocols across 284 *CL conference papers (2023-2025), discovering widespread under-reporting of critical study design details that undermine reproducibility. The findings reveal that transparency gaps in how text generation quality is assessed create ambiguity about measurement methodology, evaluator credentials, and result interpretation, prompting actionable recommendations for improved reporting standards.
AIBullisharXiv – CS AI · Jun 57/10
🧠Researchers at the Max Planck Institute compiled 100 research-level mathematics questions to benchmark large language models' reasoning capabilities. Through three evaluation stages, only 2 questions remained unsolved by advanced LLMs, indicating significant progress in AI mathematical reasoning.
AIBearisharXiv – CS AI · Jun 27/10
🧠A new study challenges claims that multimodal AI agents genuinely benefit from tool use, finding that 93-96% of problems solved with tools are also solvable without them. The research suggests these agents learn tool-calling patterns rather than actual tool-dependent capabilities, raising questions about how benchmark improvements are interpreted.
AINeutralarXiv – CS AI · Jun 27/10
🧠Mechanistic interpretability (MI) research lacks standardized auditing systems, causing conflicting findings and limiting adoption in safety-critical applications like medical AI and autonomous systems. Researchers propose a collaborative reviewing platform with continuous feedback, expert-verified guidelines, and source-based auditing to improve the field's credibility and enable broader deployment.
AIBullisharXiv – CS AI · Jun 17/10
🧠Researchers demonstrate that efficient LLM benchmarking can be substantially improved by treating it as a multiple regression problem with kernel ridge regression and applying minimum redundancy maximum relevance (mRMR) feature selection. The approach achieves lower prediction errors and faster computation than existing methods while maintaining consistency across different data splits.
AIBearisharXiv – CS AI · Jun 17/10
🧠A new arXiv paper challenges recent claims about LLM capabilities by arguing they lack scientific rigor under Popper's falsifiability principle. The authors identify methodological flaws in AI reasoning research, including opaque training data, non-reproducibility, and selection bias, then propose transparency guidelines to improve scientific integrity in LLM evaluation.
🏢 Meta
AINeutralarXiv – CS AI · May 297/10
🧠Researchers introduce PRAIB, a benchmark framework that evaluates how Large Language Models perform peer review compared to human reviewers. Analysis of 11,000 LLM-generated reviews across major AI conferences reveals significant behavioral divergences: LLM ratings show less variability, positive bias, overconfidence, and frequently miss atomic weaknesses that human reviewers catch.
AINeutralarXiv – CS AI · May 297/10
🧠Researchers propose Critique-Resilient Benchmarking, a new framework for evaluating large language models when human comprehension of tasks becomes infeasible. The method uses adversarial evaluation where answers are deemed correct if no convincing counterargument exists, allowing meaningful comparison of frontier LLMs even as they saturate traditional benchmarks.
AIBullisharXiv – CS AI · May 287/10
🧠Researchers present a unified evaluation framework for assessing LLM agentic capabilities, integrating 7 benchmarks across 24 domains with standardized testing methodology. The framework disentangles intrinsic model performance from implementation artifacts, revealing that scaffold choices and environmental volatility significantly impact benchmark results across 15 models tested.
🏢 Meta🏢 Hugging Face
AIBullisharXiv – CS AI · May 277/10
🧠Researchers conducted a 4-month case study embedding a persistent AI agent into a real academic research environment, tracking 75,671 telemetry records across 96 active days. The study reveals that persistent agents shift computational economics from cost-per-token to cost-per-artifact, with cache-dominant workflows achieving 82.9% token reuse efficiency.
AIBearisharXiv – CS AI · May 117/10
🧠Researchers demonstrate that a simple graph heuristic without machine learning matches or outperforms advanced generative recommendation systems on standard benchmarks, revealing that widely-used datasets contain structural shortcuts that don't require sophisticated modeling. The findings question whether current benchmark evaluations actually validate the advanced capabilities that modern recommendation systems claim to provide.
AIBearisharXiv – CS AI · May 77/10
🧠A research paper challenges the reliability of current AI alignment benchmarks, arguing that model-level evaluations alone cannot predict real-world deployment safety. The study finds that existing benchmarks lack user-facing verification support and that scaffold effectiveness varies dramatically across different AI models, necessitating system-level evaluation approaches rather than single performance scores.
AIBearisharXiv – CS AI · Apr 107/10
🧠Researchers discovered that GPT-4o exhibits significant daily and weekly performance fluctuations when solving identical tasks under fixed conditions, with periodic variability accounting for approximately 20% of total variance. This finding fundamentally challenges the widespread assumption that LLM performance is time-invariant and raises critical concerns about the reliability and reproducibility of research utilizing large language models.
🧠 GPT-4
AINeutralarXiv – CS AI · Mar 167/10
🧠Researchers have identified why current deepfake voice detection systems fail in real-world applications, finding that existing datasets don't account for how audio changes when transmitted through communication channels. A new framework improved detection accuracy by 39-57% and emphasizes that better datasets matter more than larger AI models for effective deepfake detection.
AINeutralarXiv – CS AI · Mar 97/10
🧠Researchers introduce AdAEM, a new evaluation algorithm that automatically generates test questions to better assess value differences and biases across Large Language Models. Unlike static benchmarks, AdAEM adaptively creates controversial topics that reveal more distinguishable insights about LLMs' underlying values and cultural alignment.
AINeutralarXiv – CS AI · Mar 57/10
🧠Researchers introduce MACC (Multi-Agent Collaborative Competition), a new institutional architecture that combines multiple AI agents based on large language models to improve scientific discovery. The system addresses limitations of single-agent approaches by incorporating incentive mechanisms, shared workspaces, and institutional design principles to enhance transparency, reproducibility, and exploration efficiency in scientific research.
AIBullisharXiv – CS AI · Mar 37/104
🧠Researchers developed a method to conduct multiple AI training experiments simultaneously within a single pretraining run, reducing computational costs while maintaining research validity. The approach was validated across ten experiments using models up to 2.7B parameters trained on 210B tokens, with minimal impact on training dynamics.
AINeutralarXiv – CS AI · Feb 277/107
🧠A research paper introduces the concept of 'vibe researching' where AI agents can autonomously execute entire research pipelines from idea to submission using specialized skills. The study analyzes how AI agents excel at speed and methodological tasks but struggle with theoretical originality and tacit knowledge, creating a cognitive rather than sequential delegation boundary in research workflows.
AIBearisharXiv – CS AI · Feb 277/104
🧠Researchers reveal a critical evaluation bias in text-to-image diffusion models where human preference models favor high guidance scales, leading to inflated performance scores despite poor image quality. The study introduces a new evaluation framework and demonstrates that simply increasing CFG scales can compete with most advanced guidance methods.
AINeutralarXiv – CS AI · Jun 235/10
🧠Researchers analyzed 73,000 reviewer trajectories from ICLR 2024-2025 to measure how author rebuttals affect peer-review scores. Using LLMs as measurement tools, they found that while rebuttals can move scores, initial review structure predicts most score movement, constraining rebuttal impact to measurable but bounded effects.
🧠 Claude🧠 Opus🧠 Gemini
GeneralNeutralarXiv – CS AI · Jun 235/10
📰A comprehensive study of 21.4 million scientific papers reveals that militaristic language in abstracts has surged 48% since 2010, correlating strongly with global conflict levels and accelerating after 2019. Experimental evidence demonstrates that war framing paradoxically undermines scientific credibility, funding support, and policy backing despite creating perceived urgency.
AIBullisharXiv – CS AI · Jun 196/10
🧠A decade-long research initiative tracking the intersection of AI and Systems Engineering has identified five critical research gaps and three evolutionary phases in the field. The study, which grew from a landmark 2020 INCOSE publication, analyzed over 2,600 papers using human-AI collaborative review to guide practitioners on AI adoption, assurance, and workforce transformation in engineering.
