FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks

Researchers introduced FinTrace, a benchmark dataset with 800 expert-annotated trajectories for evaluating how large language models perform financial tool-calling tasks. The study reveals that while frontier LLMs excel at selecting appropriate tools, they struggle significantly with information utilization and generating accurate final outputs, pointing to a critical reasoning gap that persists even after fine-tuning with preference optimization techniques.

FinTrace addresses a growing challenge in AI development: evaluating LLM performance on complex, multi-step financial tasks where tool selection alone doesn't guarantee success. The benchmark's introduction of trajectory-level metrics rather than call-level metrics represents a methodological advancement, capturing the quality of reasoning chains rather than isolated decisions. This distinction matters because financial applications demand both correct tool selection and sophisticated reasoning over retrieved information—two capabilities that don't always correlate in current models.

The research reflects broader efforts to standardize LLM evaluation in specialized domains. As financial institutions increasingly explore AI-assisted decision-making, benchmarks like FinTrace provide empirical grounding for assessing real-world readiness. The finding that 13 tested LLMs uniformly struggle with information utilization suggests this is a fundamental limitation of current architectures rather than an implementation issue.

The creation of FinTrace-Training—an 8,196-trajectory preference dataset—moves beyond diagnosis toward solutions. Direct preference optimization showed promise in suppressing failure modes compared to supervised fine-tuning, yet the persistence of end-to-end quality issues indicates that improving intermediate reasoning doesn't automatically translate to better final outputs. This gap suggests that financial tool-calling requires advances beyond trajectory optimization, potentially including better grounding mechanisms or reasoning frameworks.

For financial AI deployment, these findings validate cautious integration approaches. Organizations implementing LLM-powered financial tools should recognize that even frontier models require significant oversight and validation layers. Future work likely involves multi-stage reasoning architectures and domain-specific fine-tuning beyond preference learning.

• →Frontier LLMs select financial tools correctly but fail at reasoning over tool outputs, indicating a critical capability mismatch.
• →FinTrace's trajectory-level evaluation reveals that call-level metrics miss important reasoning quality dimensions.
• →Direct preference optimization improves intermediate reasoning metrics more effectively than supervised fine-tuning alone.
• →End-to-end answer quality remains a bottleneck even when intermediate reasoning improves, suggesting fundamental architectural limitations.
• →Financial AI deployment requires oversight mechanisms given persistent gaps between tool selection competence and reasoning effectiveness.