No Text Needed: Forecasting MT Quality and Inequity from Fertility and Metadata

Researchers demonstrate that machine translation quality can be accurately predicted without running translation systems, using only token fertility ratios, token counts, and linguistic metadata. The study achieved R² scores of 0.66-0.72 when forecasting GPT-4o translation performance across 203 languages in the FLORES-200 benchmark.

• →Translation quality can be predicted with high accuracy using only fertility ratios, token counts, and basic linguistic metadata.
• →Gradient boosting models achieved R² scores of 0.66 for translations into English and 0.72 for English translations into other languages.
• →Typological factors dominate quality predictions for translations into English, while fertility plays a larger role for diverse target languages.
• →The findings suggest translation quality is shaped by both token-level fertility and broader linguistic typology.
• →This research offers new insights for multilingual evaluation and quality estimation without running actual translation systems.