SafeHarbor: Hierarchical Memory-Augmented Guardrail for LLM Agent Safety

SafeHarbor is a new framework that enhances Large Language Model agent safety by using hierarchical memory and context-aware defense rules to prevent harmful tool use while maintaining utility on benign tasks. The system achieves 93%+ refusal rates against malicious requests while preserving 63.6% performance on legitimate tasks, addressing a critical trade-off in AI safety.

SafeHarbor addresses a fundamental challenge in deploying LLM agents at scale: the tension between security and usability. As foundation models become more capable at tool execution, they simultaneously become more dangerous when compromised. Traditional safety mechanisms rely on static rules that often over-refuse legitimate requests, creating friction for end-users and limiting practical deployment. This research proposes a dynamic solution using hierarchical memory structures that evolve based on context, allowing the system to distinguish between genuinely harmful requests and legitimate tasks that might superficially resemble them.

The technical innovation lies in combining adversarial rule generation with an information entropy-based self-evolution mechanism. Rather than requiring retraining or manual rule updates, SafeHarbor learns and adapts its defense rules through a plug-and-play architecture. The results demonstrate meaningful progress: achieving over 93% refusal rates while maintaining 63.6% utility on GPT-4o represents a significant advancement in this safety-utility trade-off that has plagued the field.

For the AI developer ecosystem, SafeHarbor offers practical implications. Organizations deploying LLM agents for production use cases can implement safety guardrails without sacrificing performance on legitimate tasks. This reduces deployment friction and accelerates adoption of autonomous agent systems across industries. The open-source availability amplifies its potential impact by lowering barriers to implementation.

Looking forward, the continuous self-evolution mechanism hints at autonomous safety systems that improve over time without human intervention. This represents a shift toward more robust, adaptive AI governance rather than static rule enforcement.

• →SafeHarbor uses dynamic hierarchical memory to balance AI safety with utility, achieving 93%+ harmful request refusal while maintaining 63.6% performance on legitimate tasks.
• →The framework employs context-aware defense rules and information entropy mechanisms that automatically optimize without requiring model retraining.
• →The training-free, plug-and-play design enables practical deployment across existing LLM agents and foundation models.
• →Results demonstrate state-of-the-art performance compared to existing defensive mechanisms that suffer from over-refusal problems.
• →Open-source availability accelerates adoption of safer LLM agent systems across the developer community.