Cost-of-Pass: An Economic Framework for Evaluating Language Models

Researchers developed a new economic framework called 'cost-of-pass' to evaluate AI language models by combining accuracy with inference costs. The study found that lightweight models are most cost-effective for basic tasks while reasoning models excel at complex problems, with costs for complex quantitative tasks roughly halving every few months.