Agentic System as Compressor: Quantifying System Intelligence in Bits

Researchers propose measuring agentic AI system intelligence through information compression, demonstrating that components like tools, retrieval, and verification reduce the bits needed to reconstruct outputs across five task domains. This analytical framework provides a quantitative method for evaluating multi-turn AI agents beyond traditional performance metrics.

The paper introduces a novel evaluation paradigm for agentic AI systems by treating intelligence as compression efficiency. Rather than assessing agents through task completion rates alone, the authors operationalize a "compression is intelligence" principle using arithmetic coding and seed coding to measure how much information reduction agentic components achieve. This approach shifts focus from what agents accomplish to how efficiently they accomplish it under fixed computational budgets.

The research emerges from a fundamental shift in AI architecture. As language models evolve from isolated prediction engines into interactive systems with tools, retrievers, and verifiers, traditional evaluation metrics struggle to capture their true capabilities. The compression framework addresses this gap by providing a unified lens across diverse domains—from chess notation to protein sequences to question answering—revealing that agentic components consistently reduce residual uncertainty. This consistency suggests the framework captures something fundamental about how agents reduce complexity.

For the AI development community, this work offers practical guidance for architectural decisions. Developers and researchers can use codelength measurements to isolate which components (tools, verifiers, retrievers) contribute most to system efficiency, informing budget allocation and design choices. The framework enables fine-grained analysis of how different components, observation quality, and computational constraints interact—insights currently unavailable through end-to-end benchmarks. The small-scale experiments serve as proof-of-concept, establishing feasibility before application to production systems.

Looking forward, this compression-based evaluation could become standard for agent research if it scales to complex, real-world tasks. The framework's information-theoretic grounding provides mathematical rigor often lacking in AI evaluation, potentially influencing how researchers prioritize agentic capabilities.

• →Compression-based metrics quantify agentic AI intelligence by measuring bits needed to reconstruct outputs, offering an alternative to traditional task-completion benchmarks.
• →Experiments across five domains consistently show that agentic components reduce codelength, validating compression as a universal measure of system efficiency.
• →The framework enables fine-grained analysis of how individual components, observation quality, and computational budgets affect residual uncertainty.
• →Information-theoretic evaluation provides mathematical rigor for comparing agent architectures under fixed resource constraints.
• →Results suggest compression-based metrics could guide practical design decisions for tool selection and budget allocation in production agents.