Query Lens: Interpreting Sparse Key-Value Features with Indirect Effects

Query Lens extends the Logit Lens technique to improve the interpretability of sparse autoencoders by analyzing both encoder key features and decoder value features, while accounting for indirect downstream effects. The research introduces the Subspace Channel Hypothesis, suggesting that neural modules process features through layer-specific subspaces, advancing understanding of how AI models process and manipulate information.

Query Lens represents a methodological advancement in neural network interpretability research, addressing a critical challenge in understanding how sparse autoencoders function. Sparse autoencoders have emerged as a promising approach to extract more human-understandable features compared to traditional neuron-level analysis, yet characterizing these features reliably remains difficult. The research tackles this by extending existing interpretability frameworks to capture a more complete picture of feature behavior throughout a neural network.

The innovation centers on jointly analyzing both inputs that activate features and outputs those features promote, then tracing indirect effects as information flows through downstream modules. This multi-level analysis goes beyond previous approaches that only captured direct effects, providing deeper insight into feature propagation and transformation. The Subspace Channel Hypothesis suggests that different layers read sparse features through distinct mathematical subspaces, implying that feature representations transform meaningfully as data moves deeper into networks.

For the AI research community, this work strengthens the foundation for mechanistic interpretability—understanding how neural networks make decisions at a granular level. As AI systems become increasingly deployed in critical applications, the ability to faithfully interpret model behavior becomes essential for debugging, verification, and ensuring alignment with intended behavior. Better interpretability tools reduce risks associated with unexplainable model outputs and can accelerate the development of more trustworthy AI systems.

The research suggests future work should validate the Subspace Channel Hypothesis across diverse architectures and scales, potentially revealing universal principles about how neural networks organize and process learned features. This could inform both theoretical understanding and practical applications in model compression, transfer learning, and safety verification.

• →Query Lens improves sparse autoencoder interpretability by analyzing both input-side key features and output-side value features simultaneously.
• →The framework accounts for indirect effects propagating through downstream neural modules, providing more complete feature characterization than previous methods.
• →The Subspace Channel Hypothesis proposes that different network layers read features through distinct mathematical subspaces.
• →Query Lens reveals coherent patterns in features that remained uninterpretable under existing Logit Lens methods.
• →Enhanced interpretability tools support the development of more trustworthy and verifiable AI systems across critical applications.