DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration

Researchers introduced DeskCraft, a new benchmark for evaluating AI desktop agents on complex, long-horizon professional workflows in creative and engineering software. The study reveals significant performance gaps, with GPT-4 achieving only 31.6% accuracy on standard tasks and 27.6% on interactive tasks requiring human collaboration, highlighting challenges in multi-step automation and proactive agent communication.

DeskCraft addresses a critical gap in AI evaluation methodology by moving beyond simplified, isolated tasks to assess how agents perform in realistic professional environments. Traditional desktop GUI benchmarks have oversimplified real-world workflows by providing complete instructions upfront and focusing on short task sequences. This new framework tests agents on tasks exceeding 50 execution steps across design, video, audio, and 3D software, more closely mimicking actual professional use.

The benchmark's innovation lies in formalizing human-in-the-loop collaboration through mid-turn and post-turn interaction protocols. Mid-turn interactions capture moments where agents seek clarification or users interrupt mid-execution, while post-turn interactions accommodate user feedback after task completion. This reflects the iterative nature of creative and engineering work, where context evolves and collaboration is essential.

The evaluation results underscore the significant distance between current AI capabilities and production-ready autonomous agents. GPT-4's performance gap between standard and interactive tasks reveals particular weaknesses in proactive communication and uncertainty handling. These shortcomings have direct implications for enterprise automation investments; organizations cannot yet rely on agents for unsupervised complex workflows without substantial human oversight.

The open-source release of DeskCraft's code, tasks, and evaluation data will likely accelerate progress in agent development by providing a standardized benchmark. However, the 27-32% performance ceiling suggests that meaningful improvements require fundamental advances in long-context reasoning, state tracking across extended workflows, and natural human-agent communication patterns rather than incremental scaling.

• →DeskCraft introduces the first major benchmark for long-horizon, human-in-the-loop desktop automation across professional creative software.
• →Current state-of-the-art models like GPT-4 achieve only 27-32% success rates on complex interactive tasks, indicating significant limitations for autonomous agent deployment.
• →The benchmark reveals persistent failures in multi-step workflow delivery and proactive agent clarification, key capabilities for practical enterprise automation.
• →Open-sourcing of DeskCraft provides the research community with standardized evaluation tools to accelerate progress on realistic AI agent capabilities.
• →Human-in-the-loop collaboration protocols in the benchmark reflect actual professional workflows, raising the bar for production-ready autonomous agent development.