#llm-experiments News & Analysis

2 articles tagged with #llm-experiments. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 116/10

🧠

Preregistration for Experiments with AI Agents

Researchers propose extending preregistration practices from human subjects research to AI agent experiments, addressing methodological vulnerabilities introduced by the ease of iterating on model selection, prompts, and experimental settings. The paper catalogs researcher degrees of freedom that make p-hacking and selective reporting easier to exploit in AI experiments while remaining difficult to detect, and calls for journals and conferences to adopt standardized preregistration templates.

AINeutralarXiv – CS AI · May 126/10

🧠

Efficient Ensemble Selection from Binary and Pairwise Feedback

Researchers present new algorithms for efficiently selecting small, high-performing ensembles of AI systems using minimal model evaluations. The work addresses both binary feedback (correct/incorrect outcomes) and pairwise feedback (preference comparisons), providing theoretical guarantees and practical query-saving methods validated through LLM experiments.

$ETH