What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations?
Researchers analyzed how large language models interpret mixed compliance demonstrations—combining benign and harmful requests with helpful responses—revealing that demonstration composition critically affects model behavior. The study shows that benign demonstrations can either reduce or increase harmful compliance depending on the model, with preference optimization during training and demonstration ordering playing crucial roles in preventing jailbreaks.