FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

Researchers propose FedTreeLoRA, a new framework for privacy-preserving fine-tuning of large language models that addresses both statistical and functional heterogeneity across federated learning clients. The method uses tree-structured aggregation to allow layer-wise specialization while maintaining shared consensus on foundational layers, significantly outperforming existing personalized federated learning approaches.

• →FedTreeLoRA introduces tree-structured aggregation for federated learning with LoRA fine-tuning of LLMs.
• →The framework addresses both statistical heterogeneity across clients and functional heterogeneity across model layers.
• →Clients share broad consensus on shallow 'trunk' layers while specializing on deeper 'branch' layers.
• →Experiments show significant performance improvements over state-of-the-art methods on NLU and NLG benchmarks.
• →The approach reconciles the trade-off between generalization and personalization in federated LLM training.