Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging

Researchers challenge the assumption that Transformers improve sleep staging through learning complex dependencies, instead revealing that random, untrained Transformers substantially boost performance by acting as adaptive smoothers. The findings suggest sleep staging relies more on architectural inductive bias than parameter learning, enabling simpler, more efficient models suitable for edge deployment in healthcare systems.

This research fundamentally questions a core assumption in modern machine learning: that Transformers succeed because they learn intricate long-range dependencies. By demonstrating that randomly initialized Transformers outperform traditional heuristic smoothing methods on sleep staging tasks without any training, the authors expose a gap between expected and actual functionality. The study reveals that sleep sequences possess strong local temporal continuity—a characteristic that favors smoothing mechanisms over complex dependency modeling.

The introduction of the Random Attention Prior Kernel (RAPK) framework provides theoretical grounding for this counterintuitive finding, showing how random self-attention balances global averaging with content-based similarity while preserving critical stage transitions. The proposed metrics—Local Smoothness Influence Index and Weighted Transition Entropy—quantify the contribution of architectural design versus learned parameters, finding that most gains derive from structure rather than training.

For healthcare and edge-computing applications, this discovery carries significant practical implications. Current Transformer-based sleep staging systems require substantial computational resources and training data. If simpler, structure-driven alternatives achieve comparable or superior performance, deployment becomes feasible on resource-constrained devices like wearables and portable monitors, democratizing sleep monitoring in clinical and consumer settings.

This work challenges the prevailing paradigm that bigger, more complex models necessarily perform better, suggesting domain-specific properties warrant deeper investigation before defaulting to state-of-the-art architectures. Future research should examine whether similar principles apply to other physiological signal classification tasks, potentially reshaping how healthcare AI systems are designed for efficiency and accessibility.

• →Random, untrained Transformers outperform traditional heuristic smoothing in sleep staging, suggesting architectural design matters more than parameter learning.
• →Sleep sequences exhibit strong local temporal continuity, making adaptive smoothing more effective than complex long-range dependency modeling.
• →The Random Attention Prior Kernel framework theoretically explains how random self-attention balances global averaging with content preservation.
• →Simpler, structure-driven models enable edge deployment of sleep staging on resource-constrained healthcare devices.
• →Research reveals most Transformer performance gains in this domain stem from inductive bias rather than learned parameters.