GOTabPFN: From Feature Ordering to Compact Tokenization for Tabular Foundation Models on High-Dimensional Data

Researchers introduce GOTabPFN, a novel approach for applying tabular foundation models to high-dimensional, low-sample-size datasets without retraining large models. The method combines Graph-guided Ordering with Local Refinement (GO-LR) and Neuro-Inspired Subunit Compression (NSC) to create compact token representations, improving prediction accuracy and stability under constrained computational budgets.

GOTabPFN addresses a fundamental challenge in machine learning: applying foundation models to tabular data with many features but few samples. Traditional large foundation models require extensive computational resources and retraining, limiting their practical deployment. This research proposes an elegant solution through intelligent feature ordering and compression mechanisms that preserve information while reducing token consumption.

The technical contribution centers on two innovations working in tandem. GO-LR provides a theoretically grounded approach to feature ordering, with connections to the weighted Minimum Linear Arrangement problem and traveling salesman path solutions. This ordering respects the underlying structure of features. The NSC unit then pools adjacent ordered features into meta-features, creating a compact representation suitable for token-constrained environments. This design respects both the mathematical properties of feature relationships and the practical limitations of modern hardware.

For the machine learning and AI community, this represents meaningful progress in making foundation models more accessible for real-world tabular prediction tasks. Many datasets in industry settings exhibit HDLSS characteristics—healthcare records, financial data, sensor readings—where foundation models have remained impractical. By enabling small foundation models to achieve better performance without architectural retraining, GOTabPFN democratizes advanced AI capabilities.

The research demonstrates improved stability and accuracy across benchmarks, suggesting the approach generalizes beyond specific dataset types. Future developments may involve integration with existing TabPFN frameworks and exploration of the feature ordering principles for other modalities. The work opens pathways for deploying foundation models in resource-constrained environments while maintaining competitive predictive performance.

• →GOTabPFN enables small tabular foundation models to handle high-dimensional, low-sample-size data without retraining large backbones.
• →Graph-guided Ordering with Local Refinement (GO-LR) provides theoretically grounded feature organization equivalent to weighted Minimum Linear Arrangement.
• →Neuro-Inspired Subunit Compression pools adjacent features into meta-features, reducing token requirements while preserving information.
• →The method improves prediction stability and accuracy under strict token budgets, making foundation models practical for real-world tabular data.
• →This approach expands foundation model applicability to healthcare, finance, and sensor data domains with HDLSS characteristics.