Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

Researchers have developed LilMoo, a 0.6-billion parameter Hindi language model trained from scratch using a transparent, reproducible pipeline optimized for limited compute environments. The model outperforms similarly sized multilingual baselines like Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that language-specific pretraining can rival larger multilingual models.

• →LilMoo is a 0.6-billion parameter Hindi language model built entirely from scratch with full transparency.
• →The model addresses linguistic inequalities in NLP by focusing on the underrepresented Hindi language.
• →A high-quality Hindi corpus called GigaLekh was created using both heuristic and LLM-based filtering methods.
• →LilMoo consistently outperforms comparably sized multilingual models like Qwen2.5-0.5B across evaluation suites.
• →The research shows that well-designed language-specific pretraining can compete with large multilingual models at sub-billion parameters.