RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

Researchers introduce RoVE (Rotary Value Embeddings), a parameter-free modification to Rotary Position Embeddings (RoPE) that makes value tokens position-sensitive in attention mechanisms. Testing on GPT-2 models demonstrates consistent improvements in few-shot learning, out-of-distribution performance, and long-context retrieval tasks.

RoVE addresses a fundamental asymmetry in modern transformer architectures where position information influences attention scoring but leaves value pathways position-blind. By rotating value embeddings alongside keys, the modification ensures that the contribution of tokens varies based on their distance from the query token, creating a more nuanced information flow. This parameter-free approach transforms RoPE attention into a form of attentive convolution, establishing theoretical connections across computer vision, robotics, and language model architectures.

The empirical validation through 124M and 354M parameter GPT-2 models reveals consistent gains across multiple evaluation dimensions. Performance improvements appear most pronounced on tasks requiring long-range aggregation, suggesting RoVE particularly benefits scenarios where distant context matters. The few-shot in-context learning gains indicate enhanced prompt understanding, while out-of-distribution perplexity improvements suggest better generalization beyond training distributions.

For the AI infrastructure sector, this advancement represents incremental progress toward more efficient and capable transformer architectures. The unification of disparate formulations across domains suggests the approach may have broad applicability beyond language models. The parameter-free nature eliminates implementation overhead while maintaining computational efficiency, reducing barriers to adoption.

Future developments should examine scaling behavior with larger models, potential integration into production systems, and whether gains persist across diverse downstream tasks. The theoretical framework connecting attention mechanisms to convolution may inspire additional architectural innovations.

• →RoVE makes transformer value pathways position-sensitive without adding parameters
• →Empirical testing shows improvements in few-shot learning and long-context retrieval tasks
• →The modification unifies attention mechanisms across computer vision, robotics, and language models
• →Parameter-free design enables straightforward integration into existing architectures
• →Strongest performance gains emerge on tasks requiring long-range token aggregation