CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios

Researchers introduce CyBiasBench, a benchmark revealing that LLM agents deployed for cybersecurity attacks exhibit inherent biases toward specific attack families regardless of prompting. The study demonstrates agents resist steering away from their preferred attack patterns, suggesting these biases are fundamental agent characteristics rather than prompt-dependent behaviors.

This research uncovers a critical vulnerability in the deployment of autonomous LLM agents for cybersecurity operations. Rather than adapting flexibly to different threat scenarios, the tested agents display consistent attack-selection biases—each gravitating toward particular attack families while largely ignoring others. This phenomenon has significant implications for both offensive and defensive cybersecurity strategies.

The finding builds on growing concerns about LLM reliability and controllability. As organizations increasingly automate security operations, understanding inherent agent limitations becomes essential. The research demonstrates that attempts to redirect agents toward alternative attack vectors through prompt engineering largely fail due to what researchers term 'bias momentum,' where agents resist distribution shifts that conflict with their trained preferences. This suggests biases are deep architectural traits rather than surface-level artifacts.

For the cybersecurity industry, these results highlight a fundamental challenge in AI-driven automation. Security teams relying on LLM agents may face unexpected limitations in attack coverage or, conversely, over-reliance on predictable patterns that defenders can anticipate. The bias phenomenon also raises important questions about AI safety in offensive contexts—autonomous systems with entrenched behavioral preferences may become difficult to control or redirect appropriately.

The release of CyBiasBench and reproducible artifacts enables further investigation into these biases across different agent architectures and training approaches. Future research should focus on whether certain architectural designs reduce bias momentum, and whether similar patterns appear in other autonomous AI applications beyond cybersecurity.

• →LLM agents exhibit attack-selection biases that concentrate efforts on narrow subsets of attack families independent of prompt variations.
• →Agent bias operates as an inherent trait resistant to steering attempts, termed 'bias momentum,' limiting the effectiveness of prompt-based redirection.
• →Bias characteristics do not correlate with attack success rates, suggesting they reflect agent preferences rather than strategic optimization.
• →The CyBiasBench benchmark provides reproducible evaluation across five agents, three targets, and four prompt conditions across ten attack families.
• →Findings raise critical concerns about controllability and predictability of autonomous LLM agents deployed in security-critical applications.