Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs

Researchers introduce Neural FOXP2, a technique that identifies and steers language-specific neurons in large language models to shift their default behavior from English to other languages like Hindi or Spanish. The method uses sparse autoencoders and spectral analysis to isolate a compact set of control circuits governing language preference, enabling safer, more targeted manipulation of multilingual model behavior.

Neural FOXP2 addresses a fundamental asymmetry in multilingual language models: despite training on diverse languages, LLMs systematically privilege English due to its dominance in pretraining data. This research mechanistically isolates the neural circuits responsible for this bias, treating language preference as a low-rank control problem rather than a distributed phenomenon scattered across model parameters.

The three-stage approach—localization via sparse autoencoders, direction identification through spectral analysis, and targeted steering—represents a meaningful advance in mechanistic interpretability. By decomposing activations into interpretable feature components and tracing selectivity patterns, researchers move beyond black-box interventions toward surgical precision. The identification of an "empirically chosen intervention window" where steering directions are strongest suggests the underlying control circuit has clear geometric structure.

This work carries implications for both model capabilities and safety. Operationally, developers could optimize models for specific regions or use cases without full retraining. More broadly, demonstrating that high-level behavioral biases stem from isolated, steerable neural circuits validates the mechanistic interpretability research agenda—if language preference is controllable through low-dimensional interventions, similar approaches might address other problematic model behaviors.

The research emphasizes "safe" steering, suggesting awareness of risks around uncontrolled model manipulation. However, the practical robustness of these interventions across different prompts, domains, and model scales remains unclear. Future work should examine whether steering holds under distribution shift and whether similar approaches generalize to other behavioral properties beyond language selection.

• →Neural FOXP2 identifies sparse, low-rank circuits governing language preference in multilingual LLMs through mechanistic interpretability techniques.
• →The method enables targeted language switching without full model retraining by steering activations in language-specific neurons across low-to-mid model layers.
• →Spectral analysis reveals dominant singular directions for language change, suggesting language bias operates through interpretable geometric structure in activation space.
• →Successfully demonstrated on Hindi and Spanish, with potential applications for region-specific model optimization and broader behavioral control.
• →Results advance mechanistic interpretability by showing high-level behavioral biases can be isolated, understood, and safely manipulated through localized interventions.