Automated evaluation of LLMs for effective machine translation of Mandarin Chinese to English

Researchers developed an automated framework to evaluate Large Language Models' effectiveness in translating Mandarin Chinese to English, comparing GPT-4, GPT-4o, and DeepSeek against Google Translate. While LLMs performed well on news translation, they showed varying results with literary texts, with DeepSeek excelling at cultural subtleties and GPT-4o/DeepSeek better at semantic conservation.

• →LLMs demonstrate strong performance in news media translation from Mandarin to English.
• →Translation quality varies significantly between different text types, with literary works presenting greater challenges.
• →DeepSeek shows superior performance in preserving cultural subtleties and grammatical rendering compared to GPT models.
• →All models struggle with maintaining cultural details, classical references, and figurative expressions in translation.
• →Automated evaluation frameworks can effectively assess translation quality without requiring time-intensive human expert reviews.