147 articles tagged with #adversarial-attacks. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers propose a formal threat model framework for evaluating distillation defenses against black-box LLM attacks, arguing that existing output perturbation defenses lack clear specifications about attacker capabilities. The work demonstrates that defense effectiveness depends heavily on assumed threat parameters, raising concerns about false security claims in deployed systems.
Researchers demonstrate a novel adversarial attack method against audio classification systems by operating in the latent space of neural audio codecs, achieving 99% attack success rates with extremely low inference latency (sub-7ms). This approach significantly outperforms existing generative and optimization-based attack methods, revealing critical vulnerabilities in real-time audio security systems like speaker verification.
Researchers have discovered that safety mechanisms in large language models operate as linear features in the output layer rather than deep semantic principles, allowing them to be manipulated or inverted through Contrastive Logit Steering. This finding reveals fundamental vulnerabilities in current alignment techniques while simultaneously suggesting a method to strengthen defenses without retraining.
Researchers demonstrate that prompt compression—a technique used to reduce LLM latency and costs—creates a new security vulnerability when processing mixed trusted and untrusted inputs. By strategically perturbing untrusted data before compression, attackers can force compressors to discard critical task information or safety guardrails, achieving 71% attack success rates through a black-box method called COMA.
A new research paper reveals critical vulnerabilities in Knowledge Editing (KE) techniques used to update facts in Large Language Models without retraining. The study demonstrates that edited knowledge is not truly erased but merely suppressed, and can be recovered through adversarial prompting, exposing fundamental flaws in current post-hoc update methods.
Researchers identified critical vulnerabilities in Large Vision-Language Models by discovering that catastrophic system collapse can be triggered by ablating just 4-5,000 neurons—a minuscule fraction of model parameters. The study reveals that these vulnerabilities are concentrated in the language backbone rather than vision components, exposing structural dependencies that challenge assumptions about model robustness.
Researchers have identified a sophisticated vulnerability in multimodal AI web agents through MIRAGE, a visual prompt injection attack that exploits trusted web platforms by embedding hidden adversarial instructions within legitimate ad slots or widgets. The attack demonstrates how constrained attackers can manipulate MLLM-based automation tools like SeeAct and OpenClaw without detection, raising critical security concerns for AI-powered browser automation systems.
Researchers identify critical vulnerabilities in LLM-solver hybrid systems where formal verification guarantees break down during the narration phase—converting solver outputs to user-readable answers. Testing five open-source models reveals adversaries can manipulate final responses through prompt injection despite underlying formal correctness, indicating safety-critical applications using AI-assisted reasoning require additional safeguards beyond solver verification.
Researchers analyzed how large language models interpret mixed compliance demonstrations—combining benign and harmful requests with helpful responses—revealing that demonstration composition critically affects model behavior. The study shows that benign demonstrations can either reduce or increase harmful compliance depending on the model, with preference optimization during training and demonstration ordering playing crucial roles in preventing jailbreaks.
Researchers demonstrate that LLM-based search engines are vulnerable to ranking manipulation attacks, where adversaries craft content to game results. Using game theory, the study reveals that reducing attack success rates can paradoxically incentivize attacks, and defensive caps may fail—highlighting the need for adaptive security strategies beyond traditional defenses.
Researchers have developed TS-LFO, an attack method that successfully bypasses copyright protection systems in AI image generation models. The technique uses two-stage optimization to restore the mapping between images and their latent representations, defeating current state-of-the-art defenses and outperforming existing copyright-stealing attacks.
Researchers demonstrate a sophisticated attack on AI safety monitoring systems where harmful behavior is distributed across many individually benign steps, encoded in temporal correlations rather than marginal statistics. Traditional per-step monitors fail by design, but temporal-correlation-based monitors can detect the attack with 79-97% accuracy, establishing a measurable detectability boundary.
Researchers demonstrate that AI-assisted peer review systems are vulnerable to simple adversarial attacks, with superficial abstract rephrasing increasing acceptance ratings by up to 1.31 points on a 10-point scale without changing underlying scientific content. The low-cost manipulation ($1, 5 minutes) reveals systemic risks in AI-mediated scientific evaluation and raises concerns about authors optimizing for algorithmic judgment rather than merit.
Researchers demonstrate Test-time Adversarial Takeover (TAKO), a novel attack that allows adversaries to remotely hijack diffusion-based robotic policies by injecting universal visual patches into camera streams. The attack achieves 100% success across multiple robotic tasks and visual encoders, revealing a critical vulnerability in vision-conditioned AI systems deployed in robotics.
Researchers have evaluated automated prompt injection attacks against large language model agents using both white-box and black-box optimization methods, finding that black-box approaches significantly outperform gradient-based techniques in realistic agentic settings. While task-universal attacks transfer effectively across domains, attacks trained on smaller models fail to generalize to frontier models like GPT-5, suggesting model-dependent vulnerabilities rather than universal exploits.
Researchers introduced MLingualFC, a benchmark revealing significant safety vulnerabilities in multilingual Vision-Language Models through flowchart-based jailbreak attacks across five languages. The study demonstrates that current VLM safety mechanisms fail to generalize across linguistic and visual modalities, with Latin script languages showing substantially higher attack success rates than non-Latin scripts like Punjabi.
Researchers introduce POISE, a novel skill-poisoning attack against LLM agents that achieves 89.3% success by embedding malicious triggers into skill instructions in ways that evade both automated detection and human inspection. The attack exploits the reliability-stealth trade-off in existing injection methods, demonstrating that current security defenses struggle to distinguish poisoned skills from legitimate ones due to high false-positive rates.
Researchers demonstrated a novel prompt-injection attack that bypasses text-based LLM defenses by encoding malicious payloads as floating-point parameters and reconstructing them as fragmented telemetry. Testing across three commercial LLM APIs showed 94.3% attack success rate against leading defenses like Prompt Guard 2, revealing a critical gap in structured-input security.
Researchers identify critical security vulnerabilities in brain-computer interface (BCI) systems connected to large language model agents, demonstrating that neural signal perturbations can manipulate tool-use authorization while evading standard safety monitors. The study establishes a formal audit framework to detect and mitigate 'brain-prompt injection' attacks, revealing that current decoder accuracy metrics fail to guarantee route safety in BCI-LLM pipelines.
Researchers demonstrate a critical vulnerability in Vision-Language Models (VLMs) used for ranking and recommendation systems through Multimodal Generative Engine Optimization (MGEO), showing that adversaries can manipulate ranking decisions by combining imperceptible image perturbations with crafted text. This attack exploits the deep cross-modal knowledge coupling within VLMs, revealing fundamental weaknesses in how these models ground and apply multimodal information.
Researchers demonstrate a new adversarial attack called Semantic Gambit that exploits Large Language Models to significantly compromise real-time Automatic Speech Recognition systems. By leveraging predictive context from LLMs, the attack achieves a 35.6% Word Error Rate—three times higher than previously documented attacks—revealing a critical vulnerability in ASR pipelines that operate under temporal constraints.
Researchers introduce EVA, an evolutionary framework that demonstrates GUI agents powered by multimodal language models are vulnerable to Environmental Injection Attacks through semantic deception rather than visual manipulation, achieving 85% attack success rates and revealing a critical security flaw in instruction-following alignment training.
Researchers have developed a new method called Controlled Latent-space Evasion that can bypass safety guardrails in language models by manipulating their internal representations more effectively than previous techniques. The attack reframes refusal suppression as an evasion problem against linear probes and achieves state-of-the-art success rates across 15 different models, highlighting a significant vulnerability in current AI safety alignment approaches.
Researchers introduce TRAP, a benchmark demonstrating that web-based AI agents are vulnerable to prompt injection attacks hidden in interface elements, with susceptibility rates ranging from 13% to 43% across frontier models. The study reveals that small contextual changes can double attack success rates, exposing systemic security weaknesses in autonomous agents performing real-world tasks like email management and professional networking.
Researchers have developed a new adversarial attack method against automatic speech recognition systems that operates in feature space rather than directly on audio waveforms, achieving significantly higher transfer rates to black-box ASR models and bypassing existing defenses. The attack uses self-supervised learning representations and vocoders to reconstruct adversarial signals, revealing critical vulnerabilities in current ASR robustness evaluation protocols.
