#generalization-gap News & Analysis

2 articles tagged with #generalization-gap. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 237/10

🧠

Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements

Researchers demonstrate that closed-loop automated machine learning systems can discover generalizable improvements in molecular property prediction by having language-model agents modify features, models, and acquire external evidence. Testing across 36 molecular endpoints reveals that while some improvements validate strongly, they don't consistently transfer to held-out test sets, highlighting critical challenges in ensuring reproducibility of AI-driven research discoveries.

AINeutralarXiv – CS AI · Jun 16/10

🧠

FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detection

Researchers introduce FBHM, a systematically curated benchmark for evaluating vision-language models on hateful meme detection across 25 rhetorical functionalities and 10 target communities. The study reveals that state-of-the-art VLMs exhibit severe generalization failures, dropping from high accuracy on standard datasets to near-random performance on FBHM, indicating they rely on dataset-specific shortcuts rather than robust multimodal reasoning. The proposed LSV (learnable steering vectors) method achieves ~30 Macro-F1 point improvements using minimal training data without degrading source-domain performance.