Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

Researchers propose a hybrid pre-training approach for language models that combines masked language modeling with a JEPA-style latent-space prediction objective, creating more semantically-aligned embeddings with better geometric properties than traditional MLM-only approaches despite achieving similar downstream accuracy.

This research addresses a fundamental limitation in modern language model training: masked language modeling, the dominant approach since BERT, tends to anchor representations to surface-level token patterns rather than deeper semantic meaning. The hybrid approach introduces a learnable balancing parameter that simultaneously optimizes both a predictive latent-space objective (borrowed from recent successes in vision) and standard MLM, forcing the model to develop richer internal representations.

The significance lies not in downstream task performance—which remains comparable—but in the geometric properties of learned embeddings. The hybrid model produces more uniform representations across different pooling strategies and exhibits richer spectral geometry, suggesting the encoder captures semantic structure more effectively. This development reflects a broader trend in representation learning where traditional benchmarks inadequately measure quality; recent work increasingly shows that embeddings with better geometric properties generalize more reliably to downstream applications, especially in zero-shot and few-shot scenarios.

For practitioners building NLP systems, this work validates that pre-training objectives fundamentally shape how models understand language, with implications extending beyond standard accuracy metrics. The finding that JEPA-style objectives reduce lexical information encoding while improving semantic capture could influence architecture choices for semantic search, retrieval systems, and specialized domain applications where semantic understanding matters more than surface-form matching.

Future research should examine whether these geometric improvements translate to better few-shot performance, cross-lingual transfer, or robustness to adversarial perturbations—areas where semantic understanding typically provides measurable advantages over lexical matching.

• →Hybrid MLM + JEPA objective produces more uniform embeddings with superior geometric properties than standard masked language modeling
• →Improved representation geometry exists despite equivalent downstream accuracy on GLUE benchmarks, suggesting standard metrics miss important quality dimensions
• →The approach reduces surface-level lexical information while enhancing semantic-to-lexical balance in learned representations
• →Learnable scalar parameter enables dynamic balancing between predictive and reconstruction objectives during training
• →Results suggest representation geometry, rather than linear-probe accuracy, may better predict real-world generalization and transfer learning performance