Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning

Researchers propose FedTSP, a federated learning method that uses pre-trained language models to generate semantically-enriched prototypes for improving model performance across heterogeneous data. The approach leverages textual descriptions of classes to preserve semantic relationships while mitigating data heterogeneity challenges in federated settings.

FedTSP addresses a fundamental limitation in Federated Prototype Learning by recognizing that maximizing inter-class distances, while improving discrimination, damages the semantic relationships essential for generalization. This represents a paradigm shift from purely geometric optimization toward semantic-aware prototype construction. The method leverages the demonstrated capability of pre-trained language models to capture rich semantic relationships from vast textual corpora—a resource unavailable in federated settings where data remains distributed.

The approach reflects broader trends in machine learning toward multimodal and cross-modal learning. By bridging the gap between visual representations on client devices and semantic understanding from language models, FedTSP demonstrates how centralized knowledge can enhance distributed learning without violating privacy constraints. The use of LLMs to generate fine-grained class descriptions and PLMs to construct textual prototypes exemplifies practical applications of foundation models beyond their traditional use cases.

For the machine learning and distributed systems communities, this work has significant implications. It suggests that semantic-preserving prototype methods could outperform purely distance-maximizing approaches in real-world federated scenarios with diverse data distributions. The introduction of trainable prompts to adapt prototypes to client-specific tasks provides a flexible mechanism for bridging modality gaps.

Looking ahead, this research direction opens questions about optimal semantic encoding, the transferability of language-derived semantics across different visual domains, and scalability to thousands of classes. Future work should explore whether different language models or semantic representations yield different performance characteristics, and whether this approach extends to other federated learning paradigms beyond prototype-based methods.

• →FedTSP uses pre-trained language models to generate semantically-rich prototypes that preserve class relationships in federated learning environments.
• →The method addresses a tradeoff between maximizing class discrimination and maintaining semantic relationships crucial for generalization.
• →Trainable prompts enable adaptation of language-derived prototypes to client-specific visual tasks across the modality gap.
• →Experimental results demonstrate FedTSP accelerates convergence while effectively mitigating data heterogeneity in federated settings.
• →The approach demonstrates practical application of foundation models in enhancing distributed learning without compromising privacy.