Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences

Researchers found that narrow finetuning of Large Language Models leaves detectable traces in model activations that can reveal information about the training domain. The study demonstrates that these biases can be used to understand what data was used for finetuning and suggests mixing pretraining data into finetuning to reduce these traces.

• →Narrow finetuning creates strong, detectable biases in LLM activations that reveal information about the training domain.
• →Simple model diffing techniques can analyze activation differences to understand finetuning objectives and generate similar content.
• →LLM-based interpretability agents perform significantly better when given access to these activation biases.
• →Mixing pretraining data into finetuning corpus can largely remove these detectable traces and biases.
• →The research raises concerns about using narrowly finetuned models as proxies for studying broader AI safety scenarios.