←Back to feed
🧠 AI⚪ NeutralImportance 4/10
Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts
🤖AI Summary
Researchers conducted a mixed-methods study evaluating an LLM-powered BPMN modeling copilot with five domain experts, revealing acceptable usability (67.2/100) but significantly lower trust levels (48.8%). The study highlights critical reliability concerns and demonstrates the need for human-centered evaluation methods beyond automated benchmarking for LLM business tools.
Key Takeaways
- →LLM-powered business process modeling tools show promise for democratizing BPMN modeling but face significant trust barriers among experts.
- →The study found acceptable perceived usability scores (67.2/100) but notably low trust ratings (48.8%) from domain experts.
- →Reliability was rated as the most critical concern by experts, scoring only 1.8 out of 5.
- →Researchers identified key issues including output quality problems, prompting difficulties, and insufficient clarifying questions from the LLM.
- →The study demonstrates that human-centered evaluation is essential to complement automated benchmarking for enterprise AI tools.
Mentioned in AI
Companies
Microsoft→
Read Original →via arXiv – CS AI
Act on this with AI
Stay ahead of the market.
Connect your wallet to an AI agent. It reads balances, proposes swaps and bridges across 15 chains — you keep full control of your keys.
Related Articles