Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation

Researchers adapted Microsoft's QuantumKatas quantum computing curriculum from Q# to Qiskit and created a 350-task benchmark with LLM evaluation infrastructure. Testing 16 language models revealed significant capability gaps, with frontier models achieving 83.1% pass rates versus 32.3% for weaker models, while highlighting that LLMs excel at implementing known algorithms but struggle with problem encoding.

This research addresses a critical gap in AI evaluation methodology by systematizing how large language models perform on quantum computing tasks. The adaptation of QuantumKatas from Q# to Qiskit represents practical infrastructure work that enables rigorous benchmarking of a domain requiring both algorithmic reasoning and specialized programming knowledge. The scale of evaluation—39,200 model runs across 16 models and 7 prompting strategies—provides statistically meaningful data on LLM capabilities in quantum computing.

Quantum computing benchmarking has traditionally focused on quantum hardware performance rather than AI systems' ability to reason about quantum problems. This work fills that void by leveraging proven pedagogical design rather than creating arbitrary tasks, ensuring the benchmark reflects meaningful learning progressions from fundamental gates through advanced algorithms like Grover's and Simon's. The 26.1 percentage point gap between frontier and open-source models indicates significant capability stratification in this domain.

The findings reveal important asymmetries in LLM reasoning: models memorize well-known algorithms but fail when required to encode classical problems into quantum solutions. This suggests current LLMs possess pattern-matching strength without deep compositional understanding of quantum-classical problem transformation. The counterintuitive chain-of-thought results—where explicit reasoning helps only reasoning-tuned models while degrading others—challenge assumptions about universal prompting strategies and suggest model-specific architectural properties influence reasoning effectiveness.

For the AI research community, this benchmark becomes a standard evaluation tool for quantum-capable LLM development. It provides actionable feedback on where models need improvement: not algorithm reproduction but creative problem formulation. The open release of benchmark, framework, and results will likely drive improved quantum reasoning in subsequent model generations.

• →QuantumKatas benchmark with 350 tasks effectively differentiates LLM quantum computing capabilities, showing 26.1pp performance gap between frontier and open-source models
• →LLMs excel at implementing known quantum algorithms (Simon's 82.1%) but struggle with problem encoding tasks (Grover SAT solving 34.4%)
• →Chain-of-thought prompting shows unexpected bimodal effects, improving performance for reasoning-tuned models but degrading others in aggregate
• →Benchmark comprises 39,200 model evaluations across 7 prompting configurations, providing statistically robust capability assessment
• →Open-source release of benchmark and evaluation framework enables standardized measurement of quantum reasoning in future LLM development