#neural-network-design News & Analysis

2 articles tagged with #neural-network-design. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 256/10

🧠

Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models

Researchers identify a critical supervision blind spot in looped language models where dense cross-entropy loss fails to control hidden-state scale variables in recurrent transitions. The study demonstrates that scale-invariant readout mechanisms like RMSNorm hide radial scaling from loss functions, allowing uncontrolled norm growth in the thousands, and proposes architectural solutions including scale-visible readouts and explicit normalization to improve model efficiency and perplexity at matched inference depths.

🏢 Perplexity

AINeutralarXiv – CS AI · May 126/10

🧠

Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers

Researchers studying one-layer Transformers discovered that architectural choices in feedforward networks (FFNs)—particularly sparse mixture-of-experts (MoE) routing—fundamentally reshape how attention mechanisms learn to compute, with sparsity rather than learned specialization driving this computational redistribution.