MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop

MacAgentBench introduces a comprehensive macOS agent benchmark with 676 tasks across 25 applications, enabling more rigorous evaluation of computer use agents (CUAs) like those deployed on Mac Mini. The study reveals that Claude Opus 4.6 on OpenClaw achieves 73.7% Pass@1, with skill libraries driving performance more than framework design, while fine-grained scoring exposes significant differences in sub-goal completion among models with similar overall scores.

MacAgentBench addresses a critical gap in AI agent evaluation by providing the first large-scale macOS-specific benchmark designed to measure real-world desktop automation capabilities. Previous benchmarks relied on binary pass/fail metrics and ignored framework augmentation—tools that modern production systems actually use. This research matters because computer use agents are moving from research labs into practical deployment, with users running systems like OpenClaw on always-on Mac Mini hardware for continuous automation.

The benchmark's 676 tasks spanning 25 applications represent genuine macOS workflows, with nearly 60% requiring both graphical interface (GUI) and command-line interface (CLI) interaction. This multi-modal requirement reflects how professionals actually work, combining visual navigation with terminal commands. The introduction of deterministic rule-based evaluation and multi-checkpoint scoring with capability annotations enables nuanced performance measurement—revealing that models with identical Pass@1 scores often complete different sub-goals, which matters for production reliability.

The experimental results carry important implications for both AI developers and enterprise users. Claude Opus 4.6's 73.7% Pass@1 represents meaningful progress, but the finding that skill libraries drive performance more than framework design suggests that the human-curated knowledge libraries remain critical. This creates an asymmetry: larger models alone don't guarantee better agent performance without domain-specific skills.

Looking ahead, this benchmark will likely become a standard evaluation tool as CUAs move into production environments. The publicly available code and data enable rapid iteration on agent design, while the fine-grained metrics provide developers with concrete targets for improvement. Future research should explore how agents transfer learned skills across applications and how to reduce the human effort required to build skill libraries.

• →MacAgentBench provides the first large-scale macOS agent benchmark with 676 tasks, enabling rigorous evaluation of computer use agents in real-world scenarios.
• →Claude Opus 4.6 on OpenClaw achieves 73.7% Pass@1, with skill libraries proving more impactful to performance than framework architecture choices.
• →Fine-grained multi-checkpoint scoring reveals that models with identical overall scores differ substantially in sub-goal completion, critical for production reliability.
• →Nearly 60% of benchmark tasks require both GUI and CLI interaction, reflecting genuine professional workflows that existing benchmarks underrepresent.
• →The research suggests that scaling model size alone has diminishing returns without accompanying improvements to agent skill libraries and framework capabilities.