Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction

Researchers evaluated how well large language models can perform formal grammar-based translation tasks using in-context learning, finding that LLM translation accuracy degrades significantly with grammar complexity and sentence length. The study identifies specific failure modes including vocabulary hallucination and untranslated source words, revealing fundamental limitations in LLMs' ability to apply formal grammatical rules to translation tasks.

This research addresses a critical bottleneck in applying large language models to low-resource language translation. Rather than testing on natural language pairs, the authors constructed a controlled experimental framework using synchronous context-free grammars to isolate the specific cognitive skill required: applying formal grammatical rules from in-context descriptions to generate correct translations. This methodology provides cleaner attribution of failure modes compared to real-world translation experiments.

The findings reveal systematic weaknesses that challenge assumptions about LLM linguistic capabilities. When grammar complexity increases or sentences lengthen, performance drops markedly—suggesting that in-context learning has meaningful constraints that scale poorly with task complexity. The error analysis showing vocabulary hallucination and recall failures indicates that models struggle with the binding problem: correctly mapping source language tokens to their target equivalents according to stated rules, rather than relying on learned statistical patterns.

For the machine translation industry, these results suggest that in-context grammar descriptions alone cannot reliably enable zero-shot translation for low-resource languages. Models may require supplementary techniques such as structured decoding, intermediate reasoning steps, or fine-tuning to reliably follow formal specifications. The morphology and script findings hint that cross-lingual transfer becomes harder when target languages have substantially different morphological structure or writing systems, limiting the universality of in-context approaches.

Future work should explore whether chain-of-thought prompting, hierarchical grammar decomposition, or architectural modifications could overcome these constraints. This research establishes clear benchmarks for evaluating whether next-generation models improve on grammar-guided transduction tasks.

• →LLM translation accuracy from formal grammars degrades significantly with increasing grammar size and sentence length
• →Morphological differences and writing script variations between languages substantially reduce model performance
• →Primary error modes include vocabulary hallucination, incorrect token recall, and failure to translate source language words
• →In-context grammatical descriptions alone appear insufficient for reliable low-resource language translation
• →Formal grammar transduction provides a controlled benchmark for measuring LLM linguistic constraint-following capabilities