#data-mixture-optimization News & Analysis

2 articles tagged with #data-mixture-optimization. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBullisharXiv – CS AI · Jun 17/10

🧠

Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training

Researchers propose DeMix, a framework that uses model merging to efficiently determine optimal data mixtures for large language model pre-training without expensive repeated training cycles. The approach decouples the search process from training costs, enabling evaluation of multiple data combinations while also releasing a 22-token dataset to support open research.

AINeutralarXiv – CS AI · Jun 96/10

🧠

Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them

Researchers identify that data mixture optimization for AI model pre-training fails at scale due to 'repetition mismatch'—when high-quality datasets are small, their repetition rates change as training budgets grow, invalidating small-scale experiments. A subsampling procedure that controls for target repetition rates enables accurate mixture prediction using only 1/16 of tokens versus traditional methods requiring 44-94% of the full budget.