Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization

Researchers introduce MEA, a new benchmark for multi-target cross-lingual summarization (MTXLS) covering 24 languages, and reveal that LLMs perform this task substantially worse than English monolingual summarization. A novel layer-wise analysis shows that translation and summarization behaviors emerge jointly in later layers rather than as separate stages, enabling a new activation steering method that improves MTXLS quality across languages.

This research addresses a genuine gap in LLM capabilities as global content consumption increasingly spans multiple languages. The MEA benchmark represents progress toward understanding how large language models handle the complex task of simultaneously summarizing and translating documents—a challenge that remains significantly harder than single-language summarization. The performance gap highlights a real limitation in current models' ability to maintain semantic fidelity while performing cross-lingual operations at scale.

The layer-wise analysis provides valuable mechanistic insights into how LLMs internally process these dual tasks. Rather than discretely handling translation followed by summarization (the intuitive pipeline approach), models appear to blend these operations throughout their processing layers, with critical work happening in deeper network regions. This finding contradicts assumptions about how transformers decompose complex linguistic tasks and suggests the brain-like emergence of sophisticated behaviors from distributed computation.

For developers building multilingual applications, these findings carry practical implications. The discovery that activation steering—using English summarization representations to guide cross-lingual generation—improves output quality across all target languages suggests new optimization techniques beyond standard fine-tuning. This approach could enhance production systems handling customer content, news distribution, or international knowledge bases without requiring language-specific models.

The research trajectory indicates a broader shift toward understanding LLM internals rather than treating them as black boxes. As models become more deployed in multilingual contexts, mechanistic understanding enables targeted improvements. Future work likely focuses on whether similar activation steering techniques apply to other cross-lingual tasks and whether insights generalize across model architectures and sizes.

• →LLMs significantly underperform at multi-target cross-lingual summarization compared to single-language tasks, creating optimization opportunities.
• →Translation and summarization emerge jointly in later transformer layers rather than as sequential processing stages.
• →Activation steering using English summarization representations consistently improves cross-lingual output quality across diverse target languages.
• →Layer-wise mechanistic analysis reveals that both task-relevant processing and errors concentrate at similar network depths.
• →Understanding LLM internals through probing methods enables inference-time improvements without retraining or architectural changes.