A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction

Researchers introduce A11y-Compressor, a framework that optimizes how AI agents interpret graphical user interfaces by transforming accessibility trees into more efficient representations. The approach reduces input tokens by 78% while simultaneously improving task success rates by 5.1 percentage points, addressing a critical bottleneck in GUI automation systems.

A11y-Compressor addresses a fundamental inefficiency in how AI agents process user interface information. Accessibility trees, while useful for encoding UI elements, generate substantial redundancy and lack spatial context—forcing language models to process verbose, linearized data that obscures the actual layout and relationships between interface components. This framework tackles that problem through modal detection, redundancy elimination, and semantic structuring, creating a more compact representation that preserves critical information while dramatically reducing computational overhead.

The practical implications extend beyond academic optimization. As AI agents become increasingly prevalent in automation workflows—from web scraping to testing to process automation—efficiency gains directly impact scalability and cost. The 78% token reduction translates to faster inference, lower API costs for services leveraging large language models, and the ability to handle more complex interfaces within existing computational budgets. The simultaneous 5.1 percentage point improvement in task success rates demonstrates that compression doesn't sacrifice capability; it actually enhances performance by presenting information more coherently.

For developers building GUI automation systems, this work provides both methodology and concrete implementation. The OSWorld benchmark validation proves real-world applicability rather than theoretical improvement. This matters particularly for enterprises deploying AI agents at scale, where token efficiency directly impacts deployment economics. The framework's modular design suggests it could integrate into existing agent architectures without fundamental redesign, lowering adoption barriers for teams already working with accessibility trees.

• →A11y-Compressor reduces input tokens to 22% of original size while improving GUI task success rates by 5.1 percentage points on average.
• →The framework addresses critical inefficiencies in how AI agents parse user interface information through compression and structural reconstruction.
• →Token reduction directly lowers computational costs and inference latency for AI systems automating GUI interactions at scale.
• →Validated performance improvements on the OSWorld benchmark indicate practical applicability beyond theoretical gains.
• →The modular pipeline design enables integration into existing automation systems without major architectural changes.