Caught in the Act(ivation): Toward Pre-Output and Multi-Turn Detection of Credential Exfiltration by LLM Agents

Researchers demonstrate that LLM agents are vulnerable to credential exfiltration attacks when sensitive data shares context windows with untrusted content, enabling indirect prompt injection. The study proposes three defense mechanisms: activation probes for pre-output detection, honeytokens with calibrated thresholds, and multi-turn leakage accounting to prevent cumulative credential theft across conversations.

This research addresses a critical vulnerability in large language model agent deployment: the co-location of sensitive credentials and user-supplied data creates exploitable pathways for attackers to extract authentication tokens and secrets through carefully crafted prompts. The threat emerges as organizations increasingly deploy LLM agents for real-world tasks that require API keys, database passwords, and other sensitive credentials to function effectively.

The vulnerability reflects a fundamental architectural tension in modern AI systems. As LLMs gain tool-use and agent capabilities, they require access to sensitive information to perform legitimate tasks. However, the same context window that enables functionality becomes an attack surface when combined with retrieval-augmented generation or user inputs. The research positions this within the broader landscape of prompt injection attacks, which have evolved from simple jailbreaks to sophisticated multi-turn strategies that exploit information leakage across conversation states.

For the AI industry, these findings underscore the maturity gap between research capabilities and production-ready security frameworks. The three proposed defenses—activation-based monitoring, format-specific honeytokens, and temporal leakage accounting—represent practical layers of defense rather than silver-bullet solutions. White-box access requirements for activation probes limit deployment flexibility, while the in-house multi-turn benchmarks lack independent validation.

Organizations deploying LLM agents for sensitive operations should prioritize implementing layered detection mechanisms rather than relying exclusively on output filtering. The research suggests that security must operate across multiple abstraction levels: internal model states, token-level canaries, and conversation-flow analysis. As agent capabilities expand, credential isolation and context management strategies warrant equivalent engineering investment to threat prevention.

• →LLM agents combining sensitive credentials with untrusted retrieved content create direct attack pathways for credential exfiltration via prompt injection.
• →Activation probes can detect credential-seeking prompts with high accuracy before tokens are output, enabling pre-emptive blocking rather than post-hoc detection.
• →Multi-turn credential theft requires cumulative tracking across conversation states, as per-turn detectors miss attacks distributed across multiple exchanges.
• →Honeytokens calibrated through conformal prediction provide format-specific detection with tunable false positive rates suitable for production deployment.
• →Current defenses require white-box access and lack standardized benchmarks, indicating security solutions remain research-stage rather than production-ready.