Instruction Following by Principled Boosting Attention of Large Language Models

Researchers developed InstABoost, a new method to improve instruction following in large language models by boosting attention to instruction tokens without retraining. The technique addresses reliability issues where LLMs violate constraints under long contexts or conflicting user inputs, achieving better performance than existing methods across 15 tasks.

• →InstABoost applies constant additive bias to instruction-key attention logits to strengthen instruction adherence in LLMs.
• →The method addresses safety and reliability risks when models violate constraints under long contexts or conflicting inputs.
• →Researchers formalized instruction following as rule-based competition between instruction rules and context-derived rules.
• →InstABoost outperformed prompting, latent steering, and prior attention steering methods across 15 evaluation tasks.
• →The technique avoids fluency collapse and instruction over-focus issues present in alternative methods.