Low-Resource Dialect Adaptation of Large Language Models: A French Dialect Case-Study

Researchers developed a cost-effective method to adapt large language models to minority dialects using continual pre-training and LoRA techniques, successfully improving Quebec French dialect performance with minimal computational resources. The study demonstrates that parameter-efficient fine-tuning can expand quality LLM access to underserved linguistic communities while updating only 1% of model parameters.

• →Continual pre-training with LoRA successfully adapted three LLMs to Quebec French dialect using very small datasets and compute budgets.
• →The method achieved improvements on minority dialect benchmarks with minimal regression on standard language performance.
• →Only 1% of model parameters needed updating to achieve meaningful dialect adaptation results.
• →Corpus composition was identified as a critical factor determining the success of dialect adaptation.
• →The first Quebec French LLMs were released on Hugging Face to support reproducibility and broader access.