113 articles tagged with #prompt-injection. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers demonstrate that reasoning traces hidden by large language models can be exposed through Reasoning Exposure Prompting (REP), a technique using shadow-model demonstrations to elicit internal reasoning through prompts. This finding challenges the security assumptions of deployed reasoning systems that intentionally conceal their internal processes from users.
Researchers demonstrate that LLM agents' decisions can be systematically manipulated through adversarial feed curation—the ordering and composition of information sources agents consume before acting. Testing on 2,785 decision rollouts across four open-source LLMs, they found feeds can shift genuinely uncertain decisions from 5% to 100% in one direction, though they cannot override firmly held model defaults, revealing a critical safety vulnerability in the upstream ranker layer rather than the model itself.
Researchers have developed an A*-inspired framework that generates obfuscated prompts capable of triggering factual errors in large language models while preserving semantic intent. The method uses a hierarchical rewrite strategy with dynamic semantic dispersion to efficiently create adversarial prompts, demonstrating higher attack success rates than existing approaches and raising urgent concerns about LLM reliability in safety-critical applications.
Researchers introduce AgentRedBench, a dynamic benchmark testing LLM agents against indirect prompt injection attacks through third-party SaaS integrations. The study reveals significant vulnerabilities across major AI models, with attack success rates up to 81%, while proposing AgentRedGuard, a specialized defense that reduces attacks to 2.4% with minimal false positives.
Researchers have identified a new jailbreak attack called Persona Attack that exploits LLMs' memory and conversation context to bypass safety mechanisms. By incrementally injecting instructions through dialogue, the attack achieves up to 95% success rates, demonstrating that accumulated memory instructions can override built-in safety alignment regardless of traditional safety training.
Researchers demonstrate that LLM agent vulnerabilities to prompt injection attacks vary dramatically depending on the injection surface used, with the same attack payload showing 96% success on one model via tool outputs but only 4% via tool descriptions. The study reveals that vulnerability is determined by model-surface interaction rather than the injection channel alone, exposing critical blindspots in current AI security evaluation methodology.
Researchers demonstrate a novel adversarial attack using genetic algorithm-based prompt injection that can deceive LLM-powered reverse engineering tools like GhidraMCP into misinterpreting binary executables. This vulnerability exploits how large language models process decompiled code through surreptitious string variable assignments, potentially allowing malware to bypass automated detection systems that rely on AI-driven analysis.
Researchers identified that indirect prompt injection attacks against ReAct AI agents succeed at dramatically different rates depending on where malicious payloads appear in tool sequences, with success rates dropping from 60% at the first tool observation to 0% at deeper positions. The study reveals that payload framing and conversation turn limits have minimal impact on attack success, making injection depth the critical vulnerability factor for AI agent systems handling real-world tasks.
Researchers reveal a critical vulnerability in LLM agents operating in local workspaces, where attackers can plant hidden prompt injections across multiple steps to gain persistent control. The new ClawTrojan benchmark demonstrates 95.5% attack success rates against GPT-5.4, while a proposed defense mechanism called DASGuard offers runtime protection by tracing and sanitizing potentially malicious control text in sensitive files.
Researchers have demonstrated that agentic AI systems used for software reverse engineering are vulnerable to prompt injection attacks embedded in executable binaries, and have developed both offensive obfuscation techniques and defensive detection methods. This research highlights critical security gaps in AI-powered code analysis tools that organizations are beginning to deploy in production environments.
Prompt injection attacks allow hackers to manipulate AI chatbots like ChatGPT, Claude, and Gemini through adversarial text inputs, potentially hijacking their behavior and outputs. OpenAI has indicated this vulnerability may be inherent to large language models and difficult to fully eliminate, raising significant security concerns for enterprises and individual users relying on these systems.
Researchers conducted the first systematic study of prompt injection attacks in real-world LLM-based resume screening, analyzing approximately 200,000 resumes from hireEZ. They found that ~1% of resumes contain hidden prompt injections, with prevalence increasing significantly over the past 1-2 years, and discovered that over 90% of injected prompts use subtle methods rather than explicit instructions.
Researchers introduce SafeSearch, an automated red-teaming framework that identifies critical vulnerabilities in LLM-based search agents by testing them against 300 adversarial cases spanning misinformation, prompt injection, and other risks. The study reveals that current search agents achieve attack success rates up to 90.5%, with common defenses like reminder prompting providing minimal protection.
Researchers propose a novel technique using early-exit mechanisms and distribution-free risk control to prevent large language models from degrading performance when exposed to harmful or irrelevant context. The approach maintains a baseline performance level (zero-shot) while selectively leveraging helpful inputs for efficiency gains, demonstrating effectiveness across multiple language tasks.
A comprehensive arXiv research review examines vulnerabilities in Large Language Models, particularly prompt injection and jailbreaking attacks, while analyzing existing defense mechanisms. The study identifies critical security gaps and proposes future research directions for safer LLM deployment across applications.
A developer embedded a prompt injection attack into the jqwik library that instructed AI coding agents to delete application output, highlighting vulnerabilities in AI-assisted development tools. The incident reveals how malicious actors can compromise open-source projects to target AI systems, creating risks for developers relying on autonomous coding agents.
Researchers demonstrate that large language model refusal behavior can be detected and exploited through intermediate layer activations before final output generation. A new attack method called Mechanistic AutoDAN leverages this discovery to achieve competitive jailbreak success rates while reducing computational time by up to 72%, raising concerns about LLM safety mechanisms.
Researchers propose the Adversarial Prompt Disentanglement (APD) framework, a defense mechanism that identifies and neutralizes malicious components in LLM inputs before processing. The system combines semantic decomposition, graph-based intent classification, and transformer-based detection to reduce harmful outputs by over 85% while maintaining model performance.
Researchers demonstrate MIRAGE, a technique that exploits vision-language model vulnerabilities in mobile GUI agents by injecting adversarial text into user-generated content regions. The attack achieves 23-30% success rates across five VLM agents without modifying apps or operating systems, revealing a critical security gap in AI-powered mobile automation that existing visual-quality defenses cannot reliably prevent.
Researchers have developed BEAP, a black-box adversarial attack that bypasses machine unlearning safeguards in text-to-image diffusion models by generating natural-language prompts that evade detection filters. The attack achieves 60% higher success rates than previous methods while remaining undetectable to safety systems, raising critical questions about the robustness of AI model safety mechanisms.
Researchers developed a testing framework to study "political plasticity"—how Large Language Models adapt their ideological responses based on user context. The study found that newer, larger LLMs reliably shift responses along economic and personal freedom axes when prompted with few-shot examples, while older models show limited adaptability, raising concerns about potential data leakage and model reliability.
Researchers have discovered WebTrap, a sophisticated prompt injection attack that can stealthily hijack browser-based AI agents during extended tasks by seamlessly blending malicious instructions with legitimate user goals. The attack maintains system usability while achieving high success rates, exposing critical vulnerabilities in autonomous agent systems that current defense mechanisms cannot adequately address.
Researchers have identified critical security vulnerabilities in multi-agent AI networks where compromised parent agents can propagate malicious instructions to spawned subagents through inherited memory. The study demonstrates how current LLM frameworks violate trust boundaries via insecure memory inheritance and weak resource controls, turning localized agent compromises into systemic network risks.
Researchers identify a critical vulnerability in agentic memory systems where Large Language Models retrieve and amplify spurious correlations from stored information, leading to erroneous reasoning in downstream decisions. The study benchmarks this risk and proposes CAMEL, a lightweight calibration method that mitigates spurious pattern reliance while maintaining performance on clean data.
Researchers have identified a critical vulnerability in LLM agents called Termination Poisoning, where adversaries inject malicious prompts to trick agents into believing tasks are incomplete, causing unbounded computation. The LoopTrap framework demonstrates this attack across 8 mainstream LLM agents with up to 25x step amplification, revealing systematic behavioral patterns that enable scalable red-teaming.
