Frequency-Domain Latent Attention Gating for Cross-Domain Token Aggregation

Researchers introduce FLaG, a novel token aggregation module that applies frequency-domain analysis via FFT to improve how transformer models combine token representations into predictions. The method shows notable performance gains on protein structure prediction and image classification tasks while maintaining competitiveness on text benchmarks.

FLaG addresses a fundamental architectural limitation in transformer-based models: the token aggregation bottleneck. Most pooling operations work exclusively in the time or spatial domain, missing potentially valuable signal patterns that emerge in the frequency spectrum. By transforming tokens using real FFT, applying learnable latent attention queries, and gating spectral components before reconstruction, FLaG introduces a cross-domain perspective that captures hierarchical information patterns previously inaccessible to standard pooling methods.

The research builds on the growing recognition that frequency-domain analysis can reveal hidden structure in sequential and spatial data. Recent advances in vision transformers and protein language models have demonstrated that many prediction tasks benefit from multi-scale representation learning. FLaG operationalizes this insight through a plug-in module, making it compatible with existing architectures like ESM2, ResNet, and RoBERTa without requiring full model retraining.

The empirical results show clearest improvements on antimicrobial peptide activity prediction and CIFAR-100 classification, domains where sample-specific spectral patterns likely encode biological or visual complexity. Ablation studies reveal that low-frequency components provide consistent signal across samples while higher frequencies capture sample-specific variations. This finding has practical implications: practitioners can tune the frequency cutoff to balance generalization against task-specific adaptation.

For the machine learning community, FLaG demonstrates that frequency-domain gating deserves broader exploration beyond standard computer vision applications. The method's interpretability through band knockouts and gate analysis provides transparency often lacking in black-box attention mechanisms. Future work should investigate whether similar frequency-based aggregation benefits other modalities like graphs or time-series data.

• →FLaG uses FFT-based frequency-domain analysis to improve token aggregation in transformer models across protein, image, and text tasks
• →Low-frequency spectral bands contribute most to predictions while higher frequencies encode sample-specific patterns
• →The module functions as a plug-in compatible with existing architectures including ESM2, ResNet, and RoBERTa
• →Performance gains are strongest on antimicrobial peptide prediction and CIFAR-100, with competitive results on text classification
• →Interpretability analysis reveals the gating mechanism acts as learnable spectral reweighting with query-wise differentiation