Short-Term Electricity Demand Forecasting for New England Using a Hybrid Transformer-XGBoost Framework with Weather, Calendar, and COVID-19 Indicators

Researchers developed a hybrid machine learning model combining Transformers and XGBoost to forecast short-term electricity demand in New England, incorporating weather, calendar, and COVID-19 data. While the hybrid approach marginally outperformed a baseline model (2.05% MAPE vs 2.21%), statistical testing revealed the improvement is not significant, and an ablation study exposed how COVID-19 features caused overfitting to pandemic-era behavioral patterns that no longer applied.

This research addresses a critical infrastructure challenge: accurate electricity demand forecasting enables grid operators to optimize resource allocation, reduce costs, and maintain reliability. The study demonstrates a sophisticated machine learning architecture that integrates multimodal data sources—meteorological observations across New England, temporal indicators, and epidemiological variables—to predict daily demand patterns. The hybrid Transformer-XGBoost framework represents a sensible architectural choice, leveraging Transformers' ability to capture complex temporal dependencies alongside XGBoost's proven effectiveness with tabular data.

However, the findings reveal a sobering limitation of modern machine learning: the Diebold-Mariano test showed the hybrid model's 4.6% improvement over the baseline is statistically indistinguishable from random noise. This challenges the assumption that architectural sophistication automatically translates to practical gains. More significantly, the ablation study uncovered a critical failure mode: COVID-19 features, which appeared beneficial during training (2020-2021), degraded test performance on post-pandemic data by 3.2%, indicating severe overfitting to transient behavioral disruptions. By mid-2022, population adaptation had eliminated the COVID-demand signal, yet the model persisted in applying learned pandemic patterns, amplifying prediction error.

This work underscores a fundamental challenge in machine learning for infrastructure: temporal validity decay. Features encoding ephemeral behavioral shifts—whether pandemic-driven or otherwise—can introduce systematic bias when conditions normalize. For practitioners deploying forecasting systems in production environments, the research suggests that feature importance rankings alone are insufficient; temporal stratification of model performance is essential to detect when learned patterns have become stale noise.

• →The hybrid Transformer-XGBoost model achieved only marginally better performance (RMSE 8,876 vs 9,304 MWh) than XGBoost alone, with statistical testing confirming the difference is not significant.
• →COVID-19 features improved training accuracy but degraded generalization to post-pandemic data by 3.2%, revealing severe overfitting to transient behavioral patterns.
• →SHAP analysis showed COVID features ranked higher in importance on post-acute test sets than during active-pandemic training, indicating the model misapplied learned patterns to new conditions.
• →Temporal validity decay emerged as a central limitation: behavioral disruptions create strong signals during crisis periods, but these features become noise once adaptation occurs.
• →The study demonstrates that architectural sophistication does not guarantee practical improvements; temporal stratification and ablation studies are critical for detecting overfitting to stale features.