LUCoS: Latent Unsupervised Context Selection for Tabular Foundation Models

Researchers introduce LUCoS, an unsupervised method for selecting training instances in tabular machine learning that uses latent embeddings rather than raw features. The approach significantly outperforms random selection across 67 datasets, addressing a critical cold-start problem in tabular foundation models like TabPFN.

LUCoS tackles a fundamental limitation in tabular machine learning: selecting which data points to label when no labeled examples exist. Traditional tabular systems struggle because raw feature spaces lack meaningful distance metrics—heterogeneous data types, mixed scales, and nonlinear relationships make simple geometric selection unreliable. The researchers demonstrate that foundation models' latent embedding spaces provide superior geometry for instance selection, enabling more effective coverage-based sampling strategies.

The core insight extends lessons from vision and language domains, where foundation models naturally produce useful embedding spaces. By leveraging an unsupervised Prior-Fitted Network to generate meaningful latent representations, LUCoS sidesteps the reliability problems plaguing original-space selection methods. Notably, their analysis shows gains split between two mechanisms: at small budeling budgets, coverage enforcement dominates; as budget increases, the quality of the representation space becomes decisive.

For the machine learning community, this work impacts practical deployment of tabular foundation models. Many real-world applications operate under strict labeling budgets, making efficient instance selection economically important. LUCoS demonstrates that sophisticated selection algorithms matter less than defining representativeness in appropriate geometric spaces. The consistency across 67 datasets and multiple evaluation metrics strengthens confidence in the approach's generalizability.

Looking forward, this research likely influences how practitioners implement TabPFN and similar models in production. The emphasis on unsupervised representation geometry may inspire similar context-selection improvements for other foundation model architectures, and the methodology could extend to semi-supervised scenarios where limited labels exist.

• →LUCoS outperforms random selection on 67 datasets by using latent embeddings instead of raw tabular features for instance selection.
• →Unsupervised representation geometry from Prior-Fitted Networks proves more reliable than original feature space for measuring data representativeness.
• →At small budgets, coverage enforcement drives performance gains; larger budgets benefit primarily from better representation spaces.
• →The approach addresses cold-start tabular learning where no labeled examples exist for selection guidance.
• →Results suggest representation quality matters more than selector algorithm sophistication for context selection.