Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting

Researchers introduce a novel predictability-aligned evaluation framework for time series forecasting that separates model performance from data's inherent unpredictability. The framework reveals that complex AI models excel with difficult-to-predict data while linear models perform comparably on more predictable tasks, suggesting current benchmark rankings conflate model capability with task difficulty.

The machine learning community's reliance on aggregate benchmark scores masks a critical problem: standard evaluation metrics cannot distinguish between genuine model improvements and easier datasets. This research introduces spectral coherence-based diagnostics that quantify forecasting difficulty independently, enabling fairer model assessment. The Spectral Coherence Predictability metric operates efficiently at O(N log N) complexity while the Linear Utilization Ratio provides frequency-resolved insights into how models exploit predictable patterns. The framework's discovery of "predictability drift"—where task difficulty fluctuates temporally—fundamentally challenges how researchers interpret model performance across time periods. Beyond methodological contributions, the finding that linear and complex models occupy different performance regimes has profound implications. In domains where data exhibits high inherent predictability (financial markets, weather patterns with strong seasonal signals), simpler models may outperform resource-intensive deep learning approaches. This efficiency insight matters significantly for practitioners deploying forecasting systems where computational costs versus accuracy trade-offs determine viability. The research undermines the narrative that newer, more complex architectures universally represent progress. Instead, it advocates for context-aware model selection based on data characteristics rather than leaderboard positioning. For cryptocurrency markets specifically, time series forecasting drives many algorithmic trading and risk management systems. If predictability varies substantially across assets and time periods—as this research suggests—current model selections may be systematically suboptimal. The framework provides tools to identify when simpler approaches suffice, potentially reducing overengineering and computational waste across fintech and quantitative finance applications.

• →Standard evaluation metrics conflate model performance with data difficulty, leading to misleading benchmark rankings and suboptimal model selection
• →Forecasting task difficulty varies significantly over time (predictability drift), requiring temporal context in model evaluation rather than aggregate scores
• →Linear models are highly effective on predictable data while complex models excel primarily on difficult-to-predict instances, suggesting context-dependent architecture choices
• →The framework's O(N log N) computational efficiency makes predictability-aligned evaluation practical for real-world deployment and continuous monitoring
• →Time series applications in crypto trading and financial forecasting could reduce computational waste by selecting appropriate complexity levels per asset