A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation

Researchers have developed a reproducible semantic benchmark for evaluating how well Large Language Models translate network intents into multivendor configurations, testing five cloud LLMs across three vendors. The study reveals that vendor effects dominate over use-case effects and highlights critical gaps in current evaluation methodologies for network automation systems.

Network automation through LLMs presents a critical infrastructure challenge: syntactically correct configurations may still fail to meet operational requirements. This paper addresses a fundamental gap in how the field evaluates LLM-based network configuration tools by introducing rigorous, reproducible testing methodology rather than relying on aggregate metrics that mask vendor-specific failure modes.

The research emerges from broader industry recognition that LLMs excel at pattern matching but struggle with semantic correctness in domain-specific applications. Network configuration demands absolute reliability—misconfigurations can cascade across infrastructure with significant consequences. Previous benchmarks lacked the rigor necessary to catch subtle failures, particularly across different vendor platforms with distinct configuration languages and operational semantics.

For enterprises deploying network automation, this work demonstrates that vendor selection significantly impacts LLM reliability more than the specific use case. The finding that repeated execution shows high dispersion and vote instability indicates current LLMs cannot be trusted for autonomous deployment without human validation. This has direct implications for DevOps teams evaluating network automation tools and AI vendors marketing configuration management solutions.

The explicit failure taxonomy and multivendor testing approach establish new standards for evaluating infrastructure-critical AI systems. As organizations increasingly rely on LLMs for network operations, this benchmarking methodology becomes essential for procurement decisions. Future work should extend this framework to other infrastructure domains where semantic correctness is non-negotiable, potentially shifting how enterprises assess AI safety and reliability before production deployment.

• →Vendor effects dominate use-case effects in LLM network configuration quality, suggesting platform choice matters more than specific deployment scenarios.
• →Repeated-run dispersion strongly predicts voting instability, indicating LLMs lack consistency for autonomous network configuration without human oversight.
• →Current aggregate metrics mask vendor-specific failure modes, particularly in platforms like Huawei VRP that expose hidden reliability issues.
• →Semantic quality and operational reliability are orthogonal properties, meaning syntactically correct configurations can still fail operationally.
• →Reproducible multivendor benchmarks are now essential standards for rigorous evaluation of LLM-based infrastructure automation systems.