DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules

Researchers introduce DiagnosticIQ, a benchmark dataset of 6,690 expert-validated questions testing whether large language models can recommend maintenance actions based on industrial sensor rules. Evaluation of 29 LLMs reveals that while frontier models perform well on standard tasks, they exhibit significant brittleness—losing 13-60% accuracy under minor perturbations and pattern-matching rather than reasoning when conditions are inverted.

DiagnosticIQ addresses a critical gap in industrial maintenance: translating sensor-triggered rules into actionable technician steps. This task demands specialized domain knowledge accumulated through years of field experience, making it an ideal stress-test for LLM reasoning capabilities. The benchmark comprises 6,690 multiple-choice questions derived from 118 rule-action pairs across 16 asset types, validated by human experts.

The research reveals a sobering reality about current LLM capabilities. While claude-opus-4-6 leads the pack with substantial margins over competitors, the benchmark's variant probing exposes fundamental weaknesses. The Pro variant demonstrates that models lose 13-60% relative accuracy when distractors are expanded—suggesting models rely on brittle pattern matching rather than robust reasoning. The Aug variant, which inverts conditions to test logical understanding, shows frontier models still select the original answer 49-63% of the time despite logical contradiction.

These findings have substantial implications for AI deployment in safety-critical industrial contexts. The bottleneck isn't raw capability but calibration and structural robustness. Models handle template-style fault detection adequately but fail when encountering structural variations or perturbations. Human practitioners achieved only 45% accuracy, confirming the benchmark's difficulty.

For industrial AI implementation, this signals the need for validation frameworks beyond standard accuracy metrics. Organizations considering LLM-based maintenance support systems must implement rigorous testing for adversarial conditions, inverted logic, and edge cases before deployment. The research suggests that current frontier models require significant additional work—likely through fine-tuning, reasoning frameworks, or hybrid human-AI systems—to achieve production-grade reliability in industrial settings.

• →DiagnosticIQ benchmark reveals frontier LLMs struggle with structural perturbations despite strong baseline performance on industrial maintenance tasks
• →Models lose 13-60% accuracy when distractors expand and exhibit pattern-matching rather than genuine reasoning under condition inversion
• →Human practitioners achieved only 45% accuracy, confirming the benchmark measures genuinely specialist knowledge requiring years of field experience
• →Deployment bottleneck is calibration and robustness rather than raw capability—models handle template tasks but break under structural variation
• →Production-grade industrial AI systems require validation frameworks beyond standard metrics to ensure safety-critical reliability