Who Gets Cited Most? Benchmarking Long-Context Numerical Reasoning on Scientific Articles

Researchers introduced SciTrek, a new benchmark for testing large language models' ability to perform numerical reasoning across long scientific documents. The benchmark reveals significant challenges for current LLMs, with the best model achieving only 46.5% accuracy at 128K tokens, and performance declining as context length increases.

• →SciTrek benchmark tests LLMs on counting, sorting, and comparing information across multiple full-text scientific articles.
• →Even the best-performing LLM achieved only 46.5% exact match accuracy at 128K token contexts.
• →Model performance degrades as context length increases, highlighting limitations in long-context reasoning.
• →LLMs particularly struggle with citation-related questions and compound logical conditions including negation.
• →The benchmark uses SQL queries over article metadata to generate verifiable questions with ground-truth answers.