DISCO: Diversifying Sample Condensation for Efficient Model Evaluation

Researchers introduce DISCO, a new method for efficiently evaluating machine learning models by selecting samples that maximize disagreement between models rather than relying on complex clustering approaches. The technique achieves state-of-the-art results in performance prediction while reducing the computational cost of model evaluation.

• →Current ML model evaluation requires thousands of GPU hours per model, creating barriers to innovation and environmental concerns.
• →DISCO selects samples based on model disagreements rather than traditional clustering methods for anchor subset selection.
• →The method uses greedy, sample-wise statistics that are conceptually simpler than global clustering approaches.
• →DISCO achieves state-of-the-art results across major benchmarks including MMLU, Hellaswag, Winogrande, and ARC.
• →Inter-model disagreement provides an information-theoretically optimal rule for greedy sample selection.