Amnesia: Adversarial Semantic Layer Specific Activation Steering in Large Language Models

Researchers have developed 'Amnesia,' a lightweight adversarial attack that bypasses safety mechanisms in open-weight Large Language Models by manipulating internal transformer states. The attack enables generation of harmful content without requiring fine-tuning or additional training, highlighting vulnerabilities in current LLM safety measures.

• →Amnesia attack can bypass existing safety mechanisms in open-weight LLMs through activation-space manipulation.
• →The attack requires no fine-tuning or additional training to generate harmful content.
• →Current reinforcement learning with human feedback measures may be insufficient for preventing misuse.
• →Open-weight LLMs are particularly vulnerable to this type of adversarial attack.
• →The research underscores the urgent need for more robust security measures in AI systems.