BlueFin: Benchmarking LLM Agents on Financial Spreadsheets

BlueFin is a new benchmark dataset that evaluates how well large language model agents perform on real-world financial spreadsheet tasks, revealing that even frontier LLMs struggle significantly with complex spreadsheet manipulation and analysis despite their advanced capabilities.

BlueFin addresses a notable gap in LLM evaluation methodology by focusing on spreadsheet-based financial tasks, an area that has received minimal research attention despite affecting hundreds of millions of users globally. The benchmark comprises 131 carefully curated tasks with 3,225 granular evaluation criteria, validated by expert human annotators to ensure high-quality assessment of complex operations that resist programmatic verification. This methodological rigor—achieving 0.826 Krippendorff's alpha agreement with expert consensus—establishes a credible foundation for benchmarking in a domain previously underexplored in AI research.

The performance results expose a critical vulnerability in current state-of-the-art LLMs. Frontier models achieve less than 50% average scores across tasks, with particular deficiencies in dynamic correctness—the ability to properly handle dependent calculations and formula chains across spreadsheets. This weakness suggests that despite advances in reasoning and code generation, LLMs lack robust understanding of spreadsheet semantics and the cascading effects of cell modifications.

The findings have significant implications for enterprise AI adoption. Financial professionals and business analysts rely heavily on spreadsheet software, and the inability of LLMs to reliably manipulate these tools limits practical deployment in high-stakes financial environments. The benchmark's public release, including the open-source evaluation harness, creates infrastructure for the research community to systematically improve agent performance on financial workflows.

Looking ahead, this work signals an emerging focus on occupational task benchmarking rather than abstract capabilities. Organizations developing financial AI assistants will need to address the dynamic correctness gap, potentially through specialized training data or architectural modifications designed specifically for spreadsheet reasoning.

• →Frontier LLMs score below 50% on BlueFin's financial spreadsheet benchmark, revealing significant performance gaps in real-world finance tasks.
• →The benchmark includes 131 complex tasks with 3,225 evaluation criteria validated by expert annotators, achieving high inter-rater agreement of 0.826 Krippendorff's alpha.
• →Dynamic correctness—handling dependent calculations and formula chains—emerges as a critical weakness in current LLM agents.
• →The open-source benchmark and evaluation framework provide infrastructure for systematic improvement in LLM spreadsheet capabilities.
• →This research highlights an underexplored domain affecting hundreds of millions of spreadsheet users globally, signaling emerging focus on occupational task benchmarking.