46 articles tagged with #red-teaming. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
An AI assistant developer conducted a security test inviting 2,000 people to attempt hacking their system, revealing vulnerabilities in AI safety and adversarial robustness. The exercise demonstrates both the challenges of securing AI systems against coordinated attacks and the importance of red-teaming in identifying real-world attack vectors before malicious actors exploit them.
Researchers present a red teaming framework using multi-role LLM architecture to systematically expose vulnerabilities in large language models, particularly unfaithfulness in responses. The approach achieved up to 7.9% improvement in attack success rates, demonstrating that architectural design choices significantly impact model safety more than parameter scaling.
Researchers applied process mining techniques to red team attack logs against large language models, revealing that standard attack success rate metrics mask critical differences in how models defend themselves. GPT-OSS 120B exhibits a near-absorbing refusal state, while Llama 3.3 70B shows multiple escape routes from refusal, with mutator effectiveness varying significantly across models.
Researchers introduce AdvGRPO, a co-training framework that enables stable joint optimization of AI attack and defense systems using reinforcement learning. The method produces transferable adversarial attacks while improving defender robustness on safety benchmarks, advancing the field of AI red teaming.
PersonaTeaming introduces a persona-driven approach to red-teaming generative AI systems, combining automated adversarial prompt generation with human-in-the-loop collaboration. The method outperforms existing automated approaches while enabling security researchers to leverage diverse perspectives and backgrounds to uncover AI model vulnerabilities more effectively.
Researchers introduce AI-Control Games, a formal mathematical framework for evaluating the safety of deploying untrusted AI systems through red-teaming exercises modeled as multi-objective stochastic games. The work demonstrates applications to language model deployment protocols, particularly Trusted Monitoring systems, offering improvements over existing empirical safety evaluation methods.
A new research study reveals that AI judges used to evaluate the safety of large language models perform poorly when assessing adversarial attacks, often degrading to near-random accuracy. The research analyzed 6,642 human-verified labels and found that many attacks artificially inflate their success rates by exploiting judge weaknesses rather than generating genuinely harmful content.
Researchers developed Q-DIG, a red-teaming method that uses Quality Diversity techniques to identify diverse language instruction failures in Vision-Language-Action models for robotics. The approach generates adversarial prompts that expose vulnerabilities in robot behavior and improves task success rates when used for fine-tuning.
Researchers introduce FERRET, a new automated red teaming framework designed to generate multi-modal adversarial conversations to test AI model vulnerabilities. The framework uses three types of expansions (horizontal, vertical, and meta) to create more effective attack strategies and demonstrates superior performance compared to existing red teaming approaches.
Researchers developed ADVERSA, an automated red-teaming framework that measures how AI guardrails degrade over multiple conversation turns rather than single-prompt attacks. Testing on three frontier models revealed a 26.7% jailbreak rate, with successful attacks concentrated in early rounds rather than accumulating through sustained pressure.
Researchers introduce CEMMA, a co-evolutionary framework for improving AI safety alignment in multimodal large language models. The system uses evolving adversarial attacks and adaptive defenses to create more robust AI systems that better resist jailbreak attempts while maintaining functionality.
OpenAI is implementing automated red teaming with reinforcement learning to protect ChatGPT Atlas from prompt injection attacks. This proactive security approach aims to discover and patch vulnerabilities early as AI systems become more autonomous and agentic.
This report details safety measures implemented before releasing a deep research system, including external red teaming exercises and frontier risk evaluations. The work follows a structured Preparedness Framework and includes built-in mitigations to address identified key risk areas.
OpenAI has released a system card detailing the safety work conducted for its new o3-mini model. The report covers safety evaluations, external red teaming exercises, and assessments under OpenAI's Preparedness Framework to ensure responsible deployment.
This document outlines a multi-layered AI safety framework based on OpenAI's established approaches, focusing on protections against prompt engineering, jailbreaks, privacy and security concerns. It details model and product mitigations, external red teaming efforts, safety evaluations, and ongoing refinement of safeguards.
OpenAI has released a system card detailing the safety evaluation process for their o1 and o1-mini models. The report covers external red teaming exercises and frontier risk assessments conducted under their Preparedness Framework before the models' public release.
The article discusses advancements in red teaming methodologies that combine human expertise with artificial intelligence capabilities. This represents a significant development in cybersecurity practices and AI safety testing approaches.
OpenAI released a system card detailing the comprehensive safety work conducted before launching GPT-4o, including external red team testing and frontier risk evaluations. The report covers safety mitigations built into the model to address key risk areas according to their Preparedness Framework.
OpenAI has announced an open call for experts to join their Red Teaming Network, focusing on improving AI model safety. The initiative seeks domain experts to help identify vulnerabilities and enhance security measures for OpenAI's AI systems.
The article title suggests content about red-teaming large language models, which involves testing AI systems for vulnerabilities and potential risks. However, no article body content was provided for analysis.
The article title suggests the introduction of a Red-Teaming Resistance Leaderboard, but no article body content was provided for analysis.
