Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance

Researchers propose an active learning framework that combines foundation model priors with smaller models to address class imbalance and label noise in real-world datasets. The method achieves over 50% annotation savings compared to existing active learning baselines while maintaining model performance across image and text domains.

This research addresses a critical pain point in machine learning: training on real-world datasets that suffer from both class imbalance and noisy labels. Traditional active learning approaches struggle when minority classes are underrepresented, as they tend to prioritize samples from majority classes. The proposed framework introduces a novel co-decision mechanism between foundation models and smaller models, leveraging the broad knowledge embedded in foundation models to guide sample selection more intelligently.

The significance extends beyond academic interest. Foundation models like BERT and vision transformers have demonstrated remarkable zero-shot and few-shot capabilities, yet their integration into active learning pipelines remains underexplored. This work fills that gap by showing how to harness foundation model priors to make imbalance-aware decisions about which samples deserve annotation effort. The systematic exploration of both label noise and class imbalance simultaneously represents a methodological advancement, as most prior work treats these challenges independently.

For practitioners and organizations, the 50% annotation savings translates directly to reduced labeling costs—a significant operational concern for machine learning teams. Annotation budgets represent substantial expenses, especially for specialized domains requiring domain expertise. The robustness to label noise is equally important, as real-world crowdsourced or user-generated labels frequently contain errors. This framework enables companies to build higher-performing models with fewer resources invested in data curation.

The work establishes foundation models as essential components in the active learning pipeline, suggesting that future annotation strategies should integrate foundation model capabilities from the outset rather than treating them as optional components. This trend likely accelerates adoption of foundation model-centric approaches across the machine learning workflow.

• →Active learning framework achieves over 50% annotation savings by leveraging foundation model priors for intelligent sample selection.
• →Method addresses dual challenges of class imbalance and label noise simultaneously across image and text domains.
• →Co-decision mechanism between foundation models and smaller models enables imbalance-aware, noise-robust sample selection.
• →Substantial cost reduction in data annotation pipelines while maintaining or improving model performance on minority classes.
• →Research establishes foundation models as critical components for efficient active learning rather than auxiliary tools.