Multi-Dimensional Spectral Geometry of Biological Knowledge in Single-Cell Transformer Representations

Researchers decoded the internal representations of scGPT, a single-cell foundation model, revealing it organizes genes into interpretable biological coordinate systems rather than opaque features. The model encodes cellular organization patterns including protein localization, interaction networks, and regulatory relationships across its transformer layers.

• →scGPT organizes genes by subcellular localization with secreted proteins at one pole and cytosolic proteins at the other.
• →The model encodes protein-protein interaction networks with perfect correlation to experimental data across confidence levels.
• →Early transformer layers preserve specific gene regulation targets while deeper layers compress into broader regulatory categories.
• →Cell-type marker genes cluster with high accuracy (AUROC = 0.851) in the model's representations.
• →The research demonstrates that biological transformers learn interpretable internal models of cellular organization.