Function Induction and Task Generalization: An Interpretability Study with Off-by-One Addition

Researchers studied how large language models generalize to new tasks through "off-by-one addition" experiments, discovering a "function induction" mechanism that operates at higher abstraction levels than previously known induction heads. The study reveals that multiple attention heads work in parallel to enable task-level generalization, with this mechanism being reusable across various synthetic and algorithmic tasks.

• →Large language models use a "function induction" mechanism that operates at higher abstraction levels than standard induction heads for task generalization.
• →Multiple attention heads work in parallel, each contributing distinct pieces to enable the +1 function induction in off-by-one addition tasks.
• →The function induction mechanism is reusable across broader task categories including shifted multiple-choice QA and base-8 addition.
• →Circuit-style interpretability techniques like path patching can effectively analyze internal model computations behind task performance.
• →The research provides insights into how composable structures within language models enable generalization to unseen tasks.