#looped-transformers News & Analysis

2 articles tagged with #looped-transformers. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 256/10

🧠

Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models

Researchers identify a critical supervision blind spot in looped language models where dense cross-entropy loss fails to control hidden-state scale variables in recurrent transitions. The study demonstrates that scale-invariant readout mechanisms like RMSNorm hide radial scaling from loss functions, allowing uncontrolled norm growth in the thousands, and proposes architectural solutions including scale-visible readouts and explicit normalization to improve model efficiency and perplexity at matched inference depths.

🏢 Perplexity

AINeutralarXiv – CS AI · Apr 146/10

🧠

Relational Preference Encoding in Looped Transformer Internal States

Researchers demonstrate that looped transformers like Ouro-2.6B encode human preferences relationally rather than independently, with pairwise evaluators achieving 95.2% accuracy compared to 21.75% for independent classification. The study reveals that preference encoding is fundamentally relational, functioning as an internal consistency probe rather than a direct predictor of human annotations.

🏢 Anthropic