SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models

SRBench introduces a comprehensive evaluation framework for Sequential Recommendation models that combines Large Language Models with traditional neural network approaches. The benchmark addresses critical gaps in existing evaluation methodologies by incorporating fairness, stability, and efficiency metrics alongside accuracy, while establishing fair comparison mechanisms between LLM-based and neural network-based recommendation systems.

SRBench represents a significant advancement in how the machine learning community evaluates recommendation systems at a time when LLM integration into diverse applications has outpaced robust evaluation frameworks. The benchmark tackles a real industry problem: existing evaluation metrics prioritize accuracy while overlooking practical requirements like fairness and computational efficiency, creating an incomplete picture of model performance in production environments.

The research identifies a fundamental fairness issue in current benchmarks—they inadvertently disadvantage LLM-based approaches by using datasets and evaluation paradigms designed for traditional neural networks. This creates artificial performance disparities that don't reflect true capabilities. By establishing a unified input paradigm through prompt engineering, SRBench enables direct, meaningful comparisons between fundamentally different architectural approaches, which is essential as LLMs increasingly augment traditional ML systems.

The prompt-extractor-coupled mechanism addresses a practical challenge unique to LLM deployment: extracting structured, task-specific answers from unstructured model outputs. This engineering solution bridges the gap between LLM flexibility and production system requirements, making LLM integration more reliable and reproducible.

The insights from evaluating 13 models reveal that LLM-based recommenders exhibit popularity bias while failing to capture nuanced item quality signals—a finding with direct implications for e-commerce, content platforms, and streaming services. This suggests LLM-based recommendation systems may require architectural modifications or specialized fine-tuning to capture quality signals effectively, opening opportunities for both researchers and practitioners to develop improved approaches.

• →SRBench establishes the first comprehensive benchmark covering accuracy, fairness, stability, and efficiency for sequential recommendation models.
• →LLM-based recommendation systems show systematic bias toward popular items while missing deeper quality signals compared to neural network baselines.
• →The unified prompt engineering paradigm enables fair performance comparison between structurally different model architectures.
• →Novel extraction mechanisms solve the practical problem of reliably converting unstructured LLM outputs into task-specific recommendations.
• →Current evaluation gaps in recommendation systems create misleading performance comparisons that don't reflect real-world production requirements.