Text2DSL: LLM-Based Code Generation for Domain-Specific Languages

Researchers introduce Text2DSL, a framework for automatically generating domain-specific language (DSL) code from natural language using large language models, validated on 4,204 Polkit security policy rules. The study demonstrates that providing structured context like BNF grammar and API specifications dramatically improves code generation accuracy to 98.6-99.4% syntactic validity across different model scales without requiring fine-tuning.

Text2DSL addresses a critical pain point in security policy management by automating the translation of natural language requirements into formal DSL code. Domain-specific languages like Polkit govern sensitive system security policies, yet current manual authoring processes require deep expertise and introduce errors that can create security vulnerabilities. This research formalizes automatic DSL generation as a distinct problem class, separate from general code generation or SQL synthesis, recognizing that DSLs have unique structural constraints requiring specialized approaches.

The PolkitBench dataset of 4,204 validated pairs provides the research community with a reproducible benchmark previously unavailable for DSL code generation. The controlled experiments reveal a fundamental insight: injecting formal specifications into prompt context acts as a robust enabling factor for high-quality LLM output. The consistency of results across models with different architectures and parameter counts—from 1.8B to 3B active parameters—suggests this finding generalizes beyond specific model families.

For enterprise security teams and developers, this work has practical implications. Reducing DSL authoring errors decreases the likelihood of policy misconfigurations that could expose systems. The approach requires no model fine-tuning, making it immediately deployable with existing LLMs through prompt engineering alone. As organizations increasingly adopt infrastructure-as-code practices, automating policy specification generation from natural language requirements could accelerate security deployment cycles while improving compliance consistency.

• →Structured context injection (BNF grammar, API specs, vocabularies) raises DSL code syntactic validity from baseline levels to 98.6-99.4%.
• →The approach works consistently across different LLM architectures without requiring model fine-tuning, making it immediately practical.
• →Text2DSL addresses a distinct problem class separate from general code generation, with domain-specific languages like Polkit having unique structural requirements.
• →Proper DSL code generation could significantly reduce security policy configuration errors in operating systems.
• →The PolkitBench dataset enables future research in automated security policy generation with 4,204 verified natural-language-to-rule pairs.