ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation

Researchers propose ZorBA, a new federated learning framework for fine-tuning large language models that reduces memory usage by up to 62.41% through zeroth-order optimization and heterogeneous block activation. The system eliminates gradient storage requirements and reduces communication overhead by using shared random seeds and finite difference methods.

• →ZorBA uses zeroth-order optimization to eliminate gradient storage at client devices during federated LLM fine-tuning
• →The framework reduces VRAM usage by up to 62.41% compared to existing federated fine-tuning methods
• →Heterogeneous block activation allows different clients to work on different transformer block subsets for improved efficiency
• →Communication overhead is reduced through shared random seeds and finite difference gradient calculations
• →An optimization algorithm jointly enhances convergence rate while minimizing memory requirements