LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

Researchers introduce LongDS, a benchmark revealing significant limitations in AI agents performing long-horizon data analysis tasks. Testing five state-of-the-art models shows best performance of only 48.45% accuracy with performance degrading by 47 points across task progression, indicating that maintaining analytical context over extended interactions remains a critical unsolved problem.

The LongDS benchmark addresses a fundamental gap in AI evaluation: most existing assessments focus on isolated or short tasks, yet real-world data analysis requires agents to maintain coherent context across dozens of sequential steps. The research demonstrates that current leading models struggle dramatically when required to track evolving analytical states through iterative workflows, with accuracy collapsing as tasks progress from early to late turns.

This findings reflect broader challenges in agentic AI development. While language models excel at point-in-time reasoning, they struggle with stateful operations requiring consistent memory and context management across extended interactions. The benchmark's construction from real Kaggle notebooks grounds the evaluation in authentic analytical patterns—counterfactual perturbation, rollback operations, and multi-state composition—rather than synthetic scenarios.

For the AI development community, these results suggest that increasing inference steps or computational budget alone cannot solve the core problem. The 52-69% of failures attributable to long-horizon errors indicate that architectural changes to how agents maintain and update internal state representations may be necessary. This has implications for enterprise applications where data analysts rely on AI assistants to execute complex, multi-step workflows.

The research signals that agentic AI systems remain immature for production use cases requiring sustained analytical reasoning. Organizations considering AI-driven data analysis tools should scrutinize how these systems handle state management across extended sessions. Future work likely focuses on novel architectures for persistent context maintenance and improved mechanisms for agents to recognize and recover from accumulated errors.

• →Best-performing models achieve only 48.45% accuracy on long-horizon data analysis tasks, indicating fundamental capability gaps
• →Performance degrades 47 percentage points from early to late turns, showing agents lose analytical context as interactions progress
• →Long-horizon errors account for 52-69% of total failures, not interaction budget limitations
• →Additional agent steps fail to improve performance, suggesting architectural rather than computational constraints
• →Current agentic AI systems remain unsuitable for production workflows requiring sustained, stateful data analysis