AgentMeter: Evaluating Model-CLI Matching for CLI-Based Local Task-Solving Agents

Researchers introduce AgentMeter, a benchmark for evaluating how language models perform with different command-line interfaces (CLIs) in local task-solving agents. The study reveals that model selection and CLI choice significantly impact performance metrics, cost, and token efficiency, demonstrating that deployment decisions require evaluating model-CLI pairs as integrated units rather than separately.

AgentMeter addresses a critical gap in AI agent evaluation by recognizing that real-world performance depends on the interaction between language models and their execution environments. Current benchmarks typically isolate model performance, but deployed agents operate through CLI interfaces that mediate context, tool outputs, and resource consumption. This research demonstrates that the same model achieves vastly different success rates and cost profiles depending on its paired CLI environment.

The benchmark's dual-tier approach—using Benchmark90 for comprehensive validation and Core30 for cost-efficient testing—enables practical trade-off analysis across 24 model-CLI configurations. The divergence in optimal configurations is particularly instructive: different deployment criteria select entirely different pairings, such as GLM-5.1 with qwen-coder for highest pass rate versus Qwen3.6+ with kimi-cli for best AMS score. This finding carries significant implications for enterprises deploying AI agents, as suboptimal pairing decisions could unnecessarily inflate operational costs or reduce reliability.

The statistical validation showing strong Spearman correlation (0.765) between Core30 and Benchmark90 results establishes credibility for cost-constrained evaluations. AgentMeter Score (AMS) provides a unified metric balancing success likelihood against resource expenditure, addressing the real-world tension between capability and cost. Organizations developing or deploying local task-solving agents should consider this framework when selecting infrastructure, as CLI choice represents a material optimization lever equivalent to model selection. Future agent development likely benefits from co-optimizing model and interface design rather than treating them as independent variables.

• →Model and CLI selection should be evaluated together as integrated deployment units, not independently
• →The same language model achieves different success rates, token efficiency, and costs under different CLI configurations
• →AgentMeter's dual-tier benchmark (Benchmark90 and Core30) enables cost-efficient evaluation with strong statistical correlation
• →Different optimization criteria select different model-CLI pairings, requiring explicit trade-off analysis for deployment decisions
• →CLI-mediated agent performance depends on how interfaces handle context, tool outputs, and terminal observations