Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

Researchers demonstrate that dense neural retrievers contain extractable sparse features matching BM25-ready vocabularies without specialized training. Sparse Autoencoders can decompose frozen dense retrievers into classical sparse retrieval components, achieving competitive or superior performance to single-vector methods while requiring no retrieval-specific supervision.

This research reveals a fundamental property of dense neural retrievers: their learned representations contain interpretable sparse structure that aligns with classical information retrieval principles. The finding that Sparse Autoencoders can extract Zipfian vocabulary distributions directly suitable for BM25 scoring suggests dense and sparse retrieval methods are not fundamentally incompatible paradigms but rather different manifestations of the same underlying learned structures.

The work builds on growing evidence that neural networks learn surprisingly interpretable latent representations. Dense retrieval has dominated recent rankings benchmarks, but practitioners often struggle with the opaqueness of vector-based methods. Latent Terms bridges this gap by showing that frozen dense retrievers already contain the information necessary for classical sparse retrieval scoring without additional training objectives or supervision. The method's effectiveness across multiple dense retriever architectures indicates this property generalizes broadly across model families.

For the information retrieval and NLP communities, this creates new opportunities for hybrid retrieval systems. Organizations invested in dense retrieval infrastructure can now extract sparse retrieval capabilities without retraining models, improving interpretability and debugging capabilities. The substantial improvements on LIMIT tasks—designed to expose single-vector retrieval failures—suggest the extracted sparse features capture complementary information that dense scoring functions miss. This enables practitioners to leverage dense retriever investments while adding classical retrieval robustness as a secondary layer, potentially improving production system reliability and search quality without architectural changes.

• →Dense retrievers implicitly learn sparse, interpretable features that match classical BM25 vocabularies without supervised training
• →Sparse Autoencoders can extract retrieval-ready latent terms from frozen dense models without requiring retrieval-specific supervision
• →The extracted sparse features achieve comparable or superior performance to single-vector scoring while improving interpretability
• →Hybrid approaches combining dense and extracted sparse retrieval substantially outperform pure single-vector methods on challenging tasks
• →This finding suggests dense and sparse retrieval methods capture complementary aspects of the same learned representation space