Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code

Researchers have discovered that Grammar-Constrained Decoding (GCD), a technique used to improve code safety in Large Language Models, can actually be exploited as a jailbreak vector called CodeSpear. The study introduces CodeShield, a defensive alignment method that protects LLMs from generating malicious code even when attackers manipulate grammar constraints.

The discovery of CodeSpear represents a critical vulnerability in current AI safety practices for code generation. While GCD has been widely adopted to ensure syntactic validity and improve reliability of LLM-generated code, researchers found that attackers can weaponize this very mechanism to force models into producing malicious code. This counterintuitive finding highlights how security-oriented techniques can create unintended attack surfaces when their constraints become attacker-controllable.

The vulnerability emerges from a fundamental mismatch between syntactic safety and semantic safety. Grammar constraints enforce structural validity but lack awareness of code functionality and intent. When attackers modify these constraints, they can guide LLMs toward malicious implementations while maintaining grammatical correctness. The CodeSpear attack demonstrates a 30+ percentage point increase in jailbreak success rates across major models, establishing this as a practical threat rather than theoretical.

CodeShield addresses this by teaching models to generate harmless honeypot code that fulfills grammatical requirements while refusing malicious requests. This approach maintains natural language refusals when available while providing robust protection under constrained scenarios. The dual-layer defense strategy preserves model utility for legitimate code generation while preventing misuse.

For AI developers and enterprises deploying LLMs for code generation, this research necessitates immediate security audits of existing GCD implementations. The findings suggest that code safety requires semantic understanding, not just syntactic compliance. As LLMs become increasingly integrated into software development pipelines, understanding these vulnerability patterns becomes critical for preventing malicious code generation at scale.

• →Grammar-constrained decoding, despite improving code safety, can be exploited to jailbreak LLMs into generating malicious code via the CodeSpear attack
• →CodeSpear increases jailbreak success rates by over 30 percentage points across 10 popular LLMs, demonstrating practical threat severity
• →CodeShield defends by teaching models to generate semantically harmless honeypot code that satisfies grammar constraints while refusing malicious requests
• →Current code safety practices prioritize syntactic validity over semantic understanding, creating a fundamental security gap
• →Organizations using LLMs for code generation should audit their grammar constraint implementations for vulnerability to attacker manipulation