newsResearchers propose FedRot-LoRA, a new framework that solves rotational misalignment issues in federated learning for large language models. The solution uses orthogonal transformations to align client updates before aggregation, improving training stability and performance without increasing communication costs.
Researchers introduce LoRA-Pre, a memory-efficient optimizer that reduces memory overhead in training large language models by using low-rank approximation of momentum states. The method achieves superior performance on Llama models from 60M to 1B parameters while using only 1/8 the rank of baseline methods.
Researchers developed a cost-effective method to adapt large language models to minority dialects using continual pre-training and LoRA techniques, successfully improving Quebec French dialect performance with minimal computational resources. The study demonstrates that parameter-efficient fine-tuning can expand quality LLM access to underserved linguistic communities while updating only 1% of model parameters.