Cost-of-Pass: An Economic Framework for Evaluating Language Models

Researchers developed a new economic framework called 'cost-of-pass' to evaluate AI language models by combining accuracy with inference costs. The study found that lightweight models are most cost-effective for basic tasks while reasoning models excel at complex problems, with costs for complex quantitative tasks roughly halving every few months.

• →The cost-of-pass framework measures the expected monetary cost of generating a correct solution from AI models.
• →Lightweight models are most cost-effective for basic quantitative tasks, while large models excel at knowledge-intensive tasks and reasoning models at complex problems.
• →Costs for complex quantitative tasks have roughly halved every few months over the past year, showing significant progress.
• →Model-level innovations are the primary drivers of cost-efficiency improvements rather than inference-time techniques.
• →Performance-oriented methods with marginal gains rarely justify their additional costs according to the economic analysis.