One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

Researchers demonstrate that instruction-tuned large language models suffer severe performance degradation when subject to simple lexical constraints like banning a single punctuation mark or common word, losing 14-48% of response quality. This fragility stems from a planning failure where models couple task competence to narrow surface-form templates, affecting both open-weight and commercially deployed closed-weight models like GPT-4o-mini.

This research exposes a fundamental vulnerability in how modern instruction-tuned language models achieve their apparent helpfulness. Rather than developing robust reasoning capabilities, these models appear to optimize for specific formatting patterns, creating brittle systems that collapse under minimal perturbation. The finding that GPT-4o-mini—a commercially deployed model—loses 31% comprehensiveness with a 99% baseline preference rate suggests this fragility extends beyond academic toy models to production systems users rely on daily.

The mechanistic analysis revealing planning failures provides crucial insight into model behavior. Base models show no systematic collapse under identical constraints, directly attributing the vulnerability to instruction tuning itself. This indicates that the training process optimizing for helpfulness inadvertently creates dependencies on surface-level formatting cues rather than genuine problem-solving strategies. The two-pass generation recovery of 59-96% of response length demonstrates the issue is architectural rather than fundamental model limitation.

For the AI industry, these findings challenge assumptions about model robustness and reliability. Organizations deploying instruction-tuned models for critical applications cannot assume consistent performance across input variations. The methodological blind spot—where standard LLM-as-judge evaluation detects only 3.5% quality drops while pairwise comparison reveals 23%—highlights evaluation frameworks may systematically underestimate model fragility. This creates risks for downstream applications and users who depend on consistent model behavior without understanding these constraints.

• →Simple lexical constraints cause instruction-tuned LLMs to lose 14-48% of response quality, with commercial models like GPT-4o-mini showing 31% comprehensiveness loss
• →The fragility stems from instruction tuning creating narrow surface-form template dependencies rather than robust reasoning capabilities
• →Base models lack systematic collapse under identical constraints, proving instruction tuning introduces the vulnerability
• →Two-pass generation (free then constrained rewriting) recovers 59-96% of lost response length, revealing planning failure mechanisms
• →Standard LLM-as-judge evaluation masks the severity of degradation, detecting only 3.5% drops where pairwise evaluation reveals 23% quality loss