The Cross-Architecture Substrate: A Domain-Transcendent, Calibration-Surviving Geometric Invariant of Modern Vision Encoders

Researchers discovered that thirteen different vision neural networks, despite being trained for distinct tasks (classification, contrast learning, image-text matching), converge on the same sixteen-dimensional geometric structure called the 'cross-architecture substrate.' This invariant structure persists across multiple visual domains and survives calibration testing, suggesting a universal representational principle in modern vision encoders that could enable new transfer learning and distillation techniques.

The discovery of a cross-architecture substrate represents a significant finding in understanding how modern vision neural networks organize information internally. The research demonstrates that despite training objectives spanning classification, contrastive learning, reconstruction, and vision-language matching, thirteen different encoder architectures develop remarkably similar top-level geometric structures. This convergence is neither coincidental nor trivial—the substrate maintains strong alignment (0.679 median Procrustes-CKA) across four visual domains and remains robust across eight domains including medical imaging, satellite data, and microscopy, indicating genuine invariance rather than artifact.

The work builds on growing evidence that neural networks discover common representational principles regardless of architecture or training regime. This extends prior findings about neural network alignment and adds specificity by identifying a precise sixteen-dimensional object that carries meaningful structure. The substrate's emergence early in training (first 10%) while accuracy continues improving suggests it represents a fundamental learning phase in vision model development. Importantly, ablation studies confirm the substrate is not explained by low-level pixel statistics, Gabor features, or random projections, pointing toward learned geometric structures.

For practitioners, the four demonstrated applications—transferability prediction, domain detection, low-shot learning, and teacher-free distillation—suggest practical utility. A sixteen-dimensional frozen probe outperforming 768-dimensional DINOv2 features on low-shot tasks hints at surprising efficiency. However, critical limitations exist: the substrate fails to transfer across modalities, provides no benefit for cross-paradigm distillation, and shows negligible correlation with transfer accuracy (rho=0.08). These boundaries clarify where the substrate applies, preventing overselling its universality.

• →Thirteen distinct vision encoders converge to the same sixteen-dimensional geometric structure despite different training objectives and architectures.
• →The cross-architecture substrate maintains strong alignment across eight visual domains (0.604 median CKA) but does not transfer across modalities.
• →A frozen sixteen-dimensional probe derived from the substrate outperforms 768-dimensional DINOv2 features by 3.78 percentage points in low-shot classification.
• →The substrate emerges in the first 10% of training and survives calibration tests, but shows negligible correlation with transfer learning performance.
• →Applications include label-free transferability prediction (3x faster than LogME), four-way domain detection (99.6% accuracy), and teacher-free knowledge distillation.