Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

Researchers introduce Jailbreak Foundry (JBF), a system that automatically converts AI jailbreak research papers into executable code modules for standardized testing. The system successfully reproduced 30 attacks with high accuracy and reduces implementation code by nearly half while enabling consistent evaluation across multiple AI models.

• →JBF addresses the challenge of rapidly evolving jailbreak techniques outpacing security benchmarks for large language models.
• →The system achieved high reproduction fidelity with only +0.26 percentage points deviation from original attack success rates.
• →JBF reduces attack-specific implementation code by nearly 50% through shared infrastructure and reusable components.
• →The platform enables standardized evaluation of 30 attacks across 10 victim models using consistent judging protocols.
• →This automation creates scalable 'living benchmarks' that can keep pace with the evolving AI security landscape.