Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability

Researchers introduce Temporal Sparse Autoencoders (T-SAEs), a new method that improves AI model interpretability by incorporating temporal structure of language through contrastive loss. The technique enables better separation of semantic from syntactic features and recovers smoother, more coherent semantic concepts without sacrificing reconstruction quality.

• →T-SAEs incorporate contrastive loss to encourage consistent activations of high-level features over adjacent tokens in sequences.
• →The method successfully disentangles semantic from syntactic features in a self-supervised manner without explicit semantic signals.
• →T-SAEs recover smoother and more coherent semantic concepts compared to traditional Sparse Autoencoders.
• →The approach addresses limitations of existing dictionary learning methods that only recover token-specific or highly local concepts.
• →Testing across multiple datasets and models shows maintained reconstruction quality while improving interpretability.