EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training

Researchers developed EstLLM, enhancing Estonian language capabilities in multilingual LLMs through continued pretraining of Llama 3.1 8B with balanced data mixtures. The approach improved Estonian linguistic performance while maintaining English capabilities, demonstrating that targeted continued pretraining can substantially improve single-language performance in multilingual models.

• →Continued pretraining with balanced data mixtures can significantly improve smaller language capabilities in multilingual LLMs without degrading primary language performance.
• →The research used Llama 3.1 8B as base model with Estonian-focused training data while maintaining English replay and technical content.
• →Post-training alignment techniques including supervised fine-tuning and preference optimization were applied to enhance instruction-following behavior.
• →Evaluation showed consistent improvements across Estonian benchmarks including linguistic competence, knowledge, reasoning, and translation quality.
• →The methodology demonstrates a viable approach for enhancing underrepresented language support in existing multilingual AI models.