UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures

Researchers introduce UR-JEPA, a novel regularization technique for Joint-Embedding Predictive Architectures that addresses representation collapse by targeting uniformly rectifiable measures rather than isotropic Gaussians. The method demonstrates superior performance on Inet10 with an 0.83 percentage-point gain over existing approaches and produces geometrically distinct embeddings with sharper spectral drops, suggesting more structured learned representations.

UR-JEPA represents a methodological advancement in self-supervised learning by reconceptualizing how neural networks should structure their learned representations. Rather than enforcing the isotropic Gaussian constraint used in LeJEPA, the approach leverages uniform rectifiability—a geometric concept from harmonic analysis—to create embeddings that naturally align with the manifold hypothesis. This theoretical shift acknowledges a fundamental tension: while preventing collapse requires dispersing representations, data typically concentrates on lower-dimensional manifolds, and forcing isotropy wastes capacity.

The technical contribution centers on implementing this geometric constraint through a Gaussian-kernel smoothed Carleson-type square function combined with Jones β-number formulations. Empirical results validate this approach across multiple datasets, with Inet10 showing the most pronounced improvement at 0.9141 accuracy with 30% lower seed variance. The spectral analysis proves particularly revealing—UR-JEPA produces a dramatic four-to-five order-of-magnitude drop in eigenvalues around dimension 20-25, contrasting sharply with LeJEPA's near-flat spectrum, indicating UR-JEPA discovers more hierarchically structured representations.

For the broader machine learning community, this work demonstrates that embedding geometry significantly influences downstream performance and stability. The consistent variance reduction across seeds suggests the method provides more reliable optimization landscapes. Though performance on Galaxy10, ImageNet-100, and EuroSAT converges between methods, UR-JEPA's geometric properties and variance characteristics position it as a more principled approach to representation learning, potentially influencing future foundation model architectures where stable, efficient embeddings drive downstream task performance.

• →UR-JEPA achieves 0.83pp improvement over LeJEPA on Inet10 with substantially lower seed variance, demonstrating more stable optimization
• →The method produces geometrically distinct embeddings with sharp spectral drops at ~20-25 dimensions, versus LeJEPA's flat spectrum
• →Uniform rectifiability provides a theoretically principled alternative to isotropic Gaussian constraints that better respects the manifold hypothesis
• →Results remain competitive across multiple datasets while maintaining superior representation structure and training stability
• →The approach reveals that embedding geometry fundamentally affects both performance and reliability in self-supervised learning systems