Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones

Researchers discovered that language models fail at balanced parentheses tasks not due to fundamental limitations, but because faulty internal mechanisms override sound ones. They developed RASteer, a steering method that amplifies reliable components, improving accuracy from 0% to nearly 100% on these tasks while maintaining general coding ability.

This research addresses a puzzling weakness in large language models: their inability to reliably handle syntactically simple tasks like balanced parentheses, despite excelling at complex coding problems. The study reveals that LM errors stem from internal competition between reliable and unreliable components rather than architectural deficiencies. Attention heads and feedforward neurons work somewhat independently, with some consistently promoting correct outputs while others introduce systematic noise. When unreliable components dominate during inference, errors emerge—a phenomenon the researchers term 'failure by interference.'

The discovery has significant implications for understanding and improving LM reliability. Rather than retraining models or redesigning architectures, RASteer offers a surgical intervention: identifying and amplifying contributions from sound components during inference. This approach boosted some models from complete failure (0% accuracy) to near-perfect performance (100%) without degrading their broader capabilities, suggesting the foundational knowledge already exists within these models—it simply needs better routing through internal mechanisms.

For the AI development community, this finding suggests that many apparent model limitations may reflect inference-time circuit conflicts rather than training failures. The broader applicability demonstrated through arithmetic reasoning tasks (20% performance gains) indicates the steering methodology could address other systematic failure modes. This has practical value for deploying smaller models more reliably in production environments where errors carry costs. Looking forward, the research opens questions about whether other coding and mathematical errors stem from similar interference patterns, and whether more sophisticated steering methods could address them without retraining.

• →Language models contain both sound and faulty mechanisms that compete during inference, with errors occurring when faulty components overshadow reliable ones.
• →RASteer methodology improves balanced parentheses accuracy from 0% to 100% on some models without retraining or impairing general coding abilities.
• →The research suggests many LM failures reflect internal circuit conflicts rather than fundamental knowledge gaps.
• →Steering techniques may offer cost-effective alternatives to retraining for improving model reliability on specific tasks.
• →Performance gains extend beyond parentheses to arithmetic reasoning, indicating broader applicability across logical reasoning domains.