GEM: Geometric Entropy Mixing for Optimal LLM Data Curation

Researchers introduce GEM (Geometric Entropy Mixing), a novel framework for optimizing LLM training data composition by treating curation as a variational problem on hyperspheres rather than relying on traditional Euclidean clustering. The method achieves up to 1.2% improvements in downstream accuracy on 1.1B-parameter models and provides a more interpretable approach to semantic data organization.

GEM addresses a fundamental challenge in large language model development: the quality and composition of training data now matters more than raw volume. Traditional data curation relies on human taxonomies and Euclidean clustering methods, both of which introduce systematic biases—human categorization often misaligns with semantic reality, while Euclidean geometry fails to account for the anisotropic properties of high-dimensional embeddings. This framework reformulates the problem geometrically, operating on hyperspheres where embedding relationships are more naturally represented.

The research builds on growing recognition within the AI community that data efficiency directly impacts model performance and training costs. As models scale, even marginal improvements in data mixing yield significant cumulative benefits across training runs and applications. GEM's introduction of the Geometric Influence Score (GIS) provides interpretability—a crucial requirement for practitioners making data composition decisions.

For the broader machine learning industry, this work has immediate practical implications. The integration with existing mixing strategies like DoReMi and RegMix demonstrates backward compatibility, lowering adoption barriers. Companies and researchers building foundational models can leverage these methods to achieve better performance without proportionally increasing compute requirements. The 1.2% accuracy improvement compounds across thousands of downstream tasks, translating to measurable value in production systems.

Future work likely involves scaling this approach to larger models and exploring how geometric data curation interacts with emerging training paradigms like synthetic data generation and curriculum learning strategies.

• →GEM reformulates data curation as a hypersphere-based variational problem, addressing limitations of Euclidean clustering and human taxonomies.
• →Achieves up to 1.2% downstream accuracy improvement on 1.1B-parameter models when integrated with existing mixing strategies.
• →Geometric Influence Score (GIS) enables interpretable, principled taxonomy generation for data organization.
• →Provable MM algorithm ensures algorithmic reliability and theoretical grounding for practical applications.
• →Teacher-student distillation enables scaling to web-scale corpora while maintaining geometric fidelity.