#experimental-methodology News & Analysis

4 articles tagged with #experimental-methodology. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

4 articles

AINeutralarXiv – CS AI · Jun 86/10

🧠

On the importance of multiple training seeds for evaluating machine unlearning

A new study reveals that evaluating machine unlearning algorithms requires multiple training seeds, not just multiple unlearning seeds from a single trained model, as unlearning performance varies significantly based on initial training conditions. This finding challenges current evaluation practices in machine unlearning research across image classification, federated learning, and large language models.

AINeutralarXiv – CS AI · Jun 16/10

🧠

FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics

Researchers introduce FML-Bench, a standardized benchmark for evaluating AI research agents that separates strategy from infrastructure, revealing that simple greedy algorithms perform comparably to complex tree-search methods. The study identifies that exploration strategy effectiveness depends on the underlying structure of optimization opportunities, with an adaptive agent demonstrating superior performance by switching strategies based on improvement stagnation detection.

AINeutralarXiv – CS AI · May 126/10

🧠

Probing Routing-Conditional Calibration in Attention-Residual Transformers

Researchers question whether routing traces in Attention-Residual transformers provide genuine evidence of improved post-hoc calibration beyond standard confidence metrics. Through rigorous statistical testing with matched controls, the study finds that routing-specific features offer minimal stable evidence of better calibration, suggesting previous claims of calibration improvements may reflect methodological artifacts rather than true model improvements.

AINeutralarXiv – CS AI · Apr 146/10

🧠

Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice

Researchers demonstrate that small-scale proxy models commonly used by AI companies to evaluate data curation strategies produce unreliable conclusions because optimal training configurations are data-dependent. They propose using reduced learning rates in proxy model training as a simple, cost-effective solution that better predicts full-scale model performance across diverse data recipes.

🏢 Meta