Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights

Researchers introduce the Mimic Score, a geometry-based metric for evaluating data quality in large datasets by measuring gradient alignment with pre-trained models. The proposed Grad-Mimic framework enables efficient data selection, reducing training steps for CLIP models by 20.7% and filtering datasets without expensive computations or validation sets.

The Mimic Score addresses a critical challenge in machine learning: identifying which samples in massive web-crawled datasets are worth training on. Current data selection methods rely on hand-crafted rules, external validation datasets, or computationally expensive influence-based approaches that don't scale efficiently. This research offers a lightweight alternative that evaluates sample utility by examining how well individual gradients align with directions induced by reference models, eliminating dependencies on validation data and reducing computational overhead significantly.

The work builds on growing recognition that training data quality matters as much as quantity. As organizations scale AI systems with increasingly large datasets, the ability to systematically identify valuable samples becomes economically important. The Grad-Mimic framework operates in two stages: dynamically re-weighting samples during training to accelerate convergence, and constructing offline filters to curate datasets beforehand. These dual approaches provide flexibility for different deployment scenarios.

The empirical results demonstrate meaningful practical impact. Achieving 20.7% fewer training steps for CLIP models translates directly to computational savings and faster model iteration cycles. The ability to filter 4.7 million samples while maintaining performance suggests that many web-scraped datasets contain substantial noise that training algorithms waste cycles processing. This efficiency gain benefits both resource-constrained organizations developing models and well-funded labs optimizing their training infrastructure.

Future developments could explore how mimic scores generalize across different model architectures, domains beyond vision, and whether the method extends to multimodal systems. Integration with existing filtering techniques suggests researchers should test combinations with other quality metrics.

• →Mimic Score enables efficient data selection by measuring gradient alignment with pre-trained models, eliminating need for validation datasets.
• →Grad-Mimic framework reduces CLIP training steps by 20.7% while maintaining performance across six image datasets.
• →Method filters datasets by identifying low-utility samples, enabling training on 4.7 million fewer samples with comparable results.
• →Geometry-based approach avoids expensive influence computations and hand-crafted heuristics that limit scalability.
• →Dual-stage framework supports both online sample re-weighting during training and offline dataset curation.