#length-extrapolation News & Analysis

2 articles tagged with #length-extrapolation. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBullisharXiv – CS AI · Jun 97/10

🧠

Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

Researchers present Polar Coordinate Position Embeddings (PoPE), an improvement to RoPE rotary position embeddings that decouples content matching from positional matching in Transformer attention mechanisms. PoPE demonstrates superior performance on language modeling, music, and genomic sequence tasks while achieving strong zero-shot length extrapolation capabilities without additional fine-tuning.

🏢 Perplexity

AINeutralarXiv – CS AI · Jun 256/10

🧠

Why Do Accumulated Transformations Extrapolate?

Researchers demonstrate that accumulated data-dependent transformations in transformer attention mechanisms enable better length extrapolation than fixed position encodings like RoPE, though performance eventually degrades at extreme context lengths. The improvement stems from learned token-dependent rotations creating finite mixing windows that suppress distant tokens while preserving near-range signals, a principle applicable across orthogonal transformations rather than specific techniques.

🏢 Perplexity