From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents

Researchers have developed Declarative Model Interface (DMI), a new abstraction layer that transforms traditional GUIs into LLM-friendly interfaces for computer-use agents. Testing with Microsoft Office Suite showed 67% improvement in task success rates and 43.5% reduction in interaction steps, with over 61% of tasks completed in a single LLM call.

• →DMI transforms existing GUIs into three declarative primitives (access, state, observation) without requiring source code modifications or APIs.
• →The system separates policy (high-level LLM planning) from mechanism (low-level DMI navigation) to improve efficiency.
• →Testing with Microsoft Office Suite demonstrated 67% higher task success rates compared to traditional GUI-based agents.
• →DMI reduced interaction steps by 43.5% and completed over 61% of successful tasks with just one LLM call.
• →The approach addresses current limitations where LLMs struggle with human-oriented interfaces that require lengthy sequences of fine-grained actions.