SpecPL: Disentangling Spectral Granularity for Prompt Learning

SpecPL introduces a novel spectral approach to prompt learning for vision-language models that decomposes visual signals into semantic low-frequency and granular high-frequency components. Using counterfactual granule supervision, the method achieves 81.51% harmonic-mean accuracy across 11 benchmarks while serving as a plug-and-play enhancement for existing text-oriented approaches.

SpecPL addresses a fundamental limitation in current vision-language model (VLM) prompt learning: the asymmetrical optimization of text tokens while visual encoders remain frozen and unable to capture fine-grained visual discrimination. The research leverages spectral decomposition through frozen VAE architectures to separate semantic invariants from granular details, a distinction that mirrors how human vision processes hierarchical information. This approach diverges from conventional prompt learning by introducing counterfactual granule training—deliberately permuting high-frequency visual signals to force explicit differentiation between semantic content and visual texture, enhancing robustness against distributional shifts.

The broader context reflects ongoing efforts to improve foundation model generalization and transfer learning efficiency. As VLMs become increasingly central to AI systems, the stability-generalization trade-off remains a critical bottleneck. SpecPL's universal plug-and-play design is particularly valuable because it retrofits existing baseline methods like CoOp and MaPLe without architectural overhauls, reducing implementation friction for practitioners.

For the AI research and development community, this work demonstrates competitive advantages in multi-benchmark evaluation while maintaining computational efficiency through frozen components. The 81.51% harmonic-mean accuracy establishes a new performance ceiling, suggesting meaningful progress on challenging generalization tasks. The released code accelerates adoption and reproducibility within the open-source ecosystem.

Looking forward, the spectral disentanglement framework could extend beyond vision-language tasks into multimodal architectures and domain adaptation scenarios where fine-grained visual discrimination proves essential.

• →SpecPL uses spectral decomposition to separate semantic low-frequency and granular high-frequency visual signals for improved prompt learning
• →Counterfactual granule training enhances model robustness by explicitly forcing discrimination between visual texture and semantic content
• →The method achieves 81.51% harmonic-mean accuracy across 11 benchmarks, establishing a new performance ceiling
• →Universal plug-and-play design allows retrofitting existing frameworks like CoOp and MaPLe without architectural modifications
• →Open-source code release accelerates adoption and reproducibility in the vision-language model research community