161 articles tagged with #llm-security. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers developed CC-BOS, a framework that uses classical Chinese text to conduct more effective jailbreak attacks on Large Language Models. The method exploits the conciseness and obscurity of classical Chinese to bypass safety constraints, using bio-inspired optimization techniques to automatically generate adversarial prompts.
Researchers discovered a new vulnerability called 'silent egress' where LLM agents can be tricked into leaking sensitive data through malicious URL previews without detection. The attack succeeds 89% of the time in tests, with 95% of successful attacks bypassing standard safety checks.
Large language models like ChatGPT face security challenges from adversarial attacks and jailbreak prompts that can bypass safety measures implemented during alignment processes like RLHF. Unlike image-based attacks that operate in continuous space, text-based adversarial attacks are more challenging due to the discrete nature of language and lack of direct gradient signals.
Researchers propose AgenticRei, a deontic policy framework for governing autonomous AI agents that goes beyond traditional access control by implementing obligations, dispensations, and conflict resolution. The system addresses critical gaps in existing policy engines like XACML and Cedar, enabling enterprises to enforce comprehensive governance constraints over LLM-driven agents that invoke tools, manipulate data, and coordinate across organizational boundaries.
Researchers propose a new empirical privacy auditing framework for fine-tuned large language models that uses synthetic canaries generated via high-temperature sampling to detect data leakage. The method also introduces a novel audit for synthetic data generated from privacy-sensitive models, revealing how model capacity and training data characteristics affect memorization risks.
Researchers introduce RecurGuard, a runtime monitoring system that defends reasoning-capable large language models against prompt injection attacks designed to exhaust generation budgets on decoy tasks. The defense detects 99% of such attacks while maintaining minimal false positives, though adaptive adversaries can partially evade detection by using topical rather than semantic attacks.
Researchers introduce CEF-Log, an LLM-based method for detecting malicious web server logs that achieves 99% F1-score using only four examples while generating forensically explainable reasoning. The approach embeds investigative methodology through structured chain-of-thought prompting, addressing the critical need for both accuracy and legal-admissible explanations in cybersecurity forensics.
SecureClaw introduces a dual-boundary security architecture designed to protect LLM agents from both unauthorized external actions and sensitive data exposure. The system uses opaque handles and a PREVIEW→COMMIT protocol to prevent language models from directly accessing secrets or executing unreviewed side effects, achieving zero attack success rates on major security benchmarks.
Researchers propose a new observability framework for tracking delegated execution in AI agent systems, addressing a critical gap where audit logs fail to distinguish which delegation scope authorized specific actions. The solution uses a lightweight gateway and information model to enable forensic reconstruction of agent activities across heterogeneous tools without relying on unreliable time-window correlation.
GuardNet, an ensemble-based detection system using shallow neural networks, demonstrates competitive performance in identifying prompt injection and jailbreak attacks on large language models while operating at 50ms latency suitable for production deployment. Although larger LLMs outperform it on some benchmarks, GuardNet achieves strong results (0.747 AUROC) with significantly lower computational overhead, challenging the assumption that adversarial robustness requires massive model scale.
Researchers introduce PropMe, a framework that distinguishes between LLMs' capability to leak training data when directly attacked versus their propensity to do so during normal use. Testing on open models reveals a significant gap: while models can be forced to reproduce training data through adversarial prompts, they rarely do so voluntarily, suggesting memorization risk is lower in practical deployment than worst-case evaluations suggest.
Researchers present Symbolicate-Enrich-Sample, a batch pipeline that uses LLM assistance to prioritize vulnerability research targets across millions of Windows functions. By combining symbol recovery, structural analysis, and language model reasoning, the system reduces 7.2 million functions to a manageable 22,000-function shortlist for security analysis.
Researchers propose Agent Guide, a behavioral watermarking framework designed to trace and protect intelligent agents deployed in digital ecosystems by embedding watermarks in high-level decision patterns rather than token sequences. The framework addresses vulnerabilities in traditional LLM watermarking by decoupling agent behavior from specific actions, enabling reliable watermark detection while maintaining natural execution patterns.
Researchers have developed mcp-attested, a security extension to the Model Context Protocol that enables safe integration of third-party tool servers with LLM agents through cryptographic attestation, allowlists, and audit logging. The mechanism addresses critical trust gaps in how AI agents interact with external services without modifying existing protocols, establishing a framework that could become an MCP standard.
Researchers introduce CyberTeam, a benchmark framework that standardizes how Large Language Models assist cybersecurity blue teams in threat hunting. The framework integrates 30 tasks and 9 operational modules into a structured workflow, showing that guided, modularized approaches significantly outperform open-ended reasoning strategies in real-world threat detection scenarios.
Researchers introduce Honeyval, a comprehensive evaluation framework for testing LLM-powered HTTP honeypots against AI-driven attackers. The framework addresses scalability and reproducibility gaps in existing honeypot evaluations, revealing that LLM-based honeypots substantially outperform rule-based systems in engagement duration while remaining difficult to detect, though trade-offs exist between interaction length and detection evasion.
Researchers demonstrate that Large Language Model-based multi-agent systems are vulnerable to coordinated attacks where malicious agents collaborate to spread misinformation more effectively than independent attackers. They propose STAR, a defense mechanism using sentence-level analysis that recovers 36.76% of lost performance by identifying and correcting misleading information in agent communications.
Researchers present ChainCaps, a runtime safety framework that prevents tool-using AI agents from exploiting composed services through 'permission laundering'—where an agent passes intermediate results through multiple tools to achieve unauthorized outcomes. The system uses capability budgets that propagate through tool chains via intersection, reducing attack success rates from 25-68% to 0-4.8% while maintaining 96-100% benign task completion across frontier models.
Researchers introduce Cordon-MAS, a new defense framework against poisoning attacks on retrieval-augmented generation (RAG) systems. The framework reduces attack success rates by 92.4% by enforcing information-flow control that prevents synthesis agents from directly accessing untrusted evidence, addressing a critical vulnerability in AI systems used for high-stakes applications.
Researchers propose a dynamic defense mechanism for Multi-Agent Systems that identifies and isolates malicious agents by computing each agent's contribution to final outputs through backward propagation. The method addresses a critical vulnerability where adversarial agents can inject false information that spreads through agent networks, improving security for LLM-based multi-agent applications.
Researchers introduce Shadow Unlearning, a privacy-preserving machine unlearning method that removes training data influence from LLMs without exposing sensitive information to attacks. The Neuro-Semantic Projector Unlearning (NSPU) framework achieves this while maintaining model performance and is 10x more computationally efficient than existing approaches.
Researchers introduce VulTriage, an LLM-based framework that enhances vulnerability detection in source code through triple-path context augmentation combining control flow analysis, vulnerability knowledge retrieval, and semantic summarization. The approach achieves state-of-the-art results on benchmark datasets and demonstrates strong generalization to low-resource scenarios.
Researchers developed ARSM-Agent, a security-enhanced framework for medical decision-making AI systems that defends against adversarial attacks through multi-module validation. The system reduces attack success rates to 8.7% while maintaining 91% knowledge consistency, demonstrating significant improvements over existing baseline approaches.
Researchers evaluated prompt-injection defenses for educational LLM tutors, revealing inherent trade-offs between security, usability, and speed. A multi-layer safeguard pipeline achieved 46.34% attack bypass with zero false positives and 2.50ms latency, while competing systems like NeMo Guardrails eliminated bypasses but suffered 16.22% false positive rates and 1.3-second delays.
Researchers have successfully demonstrated methods to remove watermarks from large language model outputs through various text manipulation techniques including paraphrasing and machine translation. The study reveals that current watermarking schemes designed to prevent misuse of LLMs are vulnerable to attack, raising questions about their effectiveness as security measures.
