Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data

Researchers demonstrate how CLIP-style vision-language models acquire left-right spatial understanding through a controlled 1D testbed, revealing that label diversity drives generalization more than layout diversity. Mechanistic analysis shows that interactions between positional and token embeddings create horizontal attention gradients that break left-right symmetry, providing insights into how Transformer-based models develop relational competence.

This research addresses a fundamental gap in understanding how vision-language models like CLIP develop spatial reasoning capabilities. By creating a highly controlled experimental environment with synthetic 1D image-text pairs, the authors isolate the mechanisms underlying spatial relation learning, moving beyond black-box observations to reveal the computational processes at work.

The significance lies in its mechanistic clarity. Rather than merely confirming that models learn spatial relations, the study decomposes attention patterns to show precisely how positional embeddings interact with token embeddings to generate directional biases. This horizontal attention gradient—the mechanism breaking left-right symmetry—is not an obvious feature and would be difficult to discover without systematic ablation studies. The finding that label diversity outweighs layout diversity in driving generalization challenges intuitions about what improves model robustness.

For the broader AI development community, this work contributes to interpretability research, a field increasingly critical as models are deployed in real-world applications. Understanding how models acquire relational understanding enables better training strategies and more predictable behavior. The controlled testbed approach itself offers a replicable methodology for probing other fundamental model capabilities.

The research also has implications for model reliability. If spatial understanding emerges through specific embedding interactions, this suggests vulnerabilities or opportunities for improving robustness through targeted training data composition. Future work might explore whether similar mechanisms explain spatial reasoning in larger, more complex models, or whether scaling introduces qualitatively different learning pathways.

• →CLIP-style contrastive training successfully learns left-right spatial relations through positional-token embedding interactions that create horizontal attention gradients
• →Label diversity is the primary driver of spatial generalization, more important than increasing layout diversity in training data
• →Attention decomposition reveals that breaking left-right symmetry depends critically on positional embedding contributions, which can be empirically validated through ablation
• →The controlled 1D testbed approach provides a replicable methodology for mechanistically understanding how vision-language models acquire relational competence
• →Understanding these mechanisms could inform better training strategies and improve predictability of spatial reasoning in larger-scale vision-language models