Energy Forecasting Model Benchmarking
Limitations of Traditional Benchmarks
Energy forecasting models are usually compared on backtests and fixed public datasets. Backtests can use data that did not yet exist when the forecast would have been made, and a fixed test set, reused across studies and possibly contained in the pretraining data of foundation models, invites overfitting to the benchmark. EnergyBench evaluates every entry live instead.
Live Evaluation Mechanism
Each day at 11:30 CET/CEST, thirty minutes before the day-ahead (SDAC) gate closes, every model forecasts the next day's price, wind and solar generation and load on the 15-minute grid for six bidding zones (BE, NL, FR, DE-LU, DK1, DK2).
Unlike existing live benchmarks, EnergyBench runs the models itself, on one information set that it freezes at the cut-off and publishes, including three numerical weather prediction models, so that foundation models can be compared with and without the same weather covariates under the actual market deadline.
Errors are pooled per series against a seasonal-naive baseline, and models are ranked by a Bradley-Terry rating on the Elo scale with a day-bootstrap interval.
Performance Findings
- baselines
- linear and gradient-boosted models on the frozen weather
- eighteen time-series foundation models from eleven families
- ensembles
- the grid operators' own forecasts
Over the first thirty-two live days, the covariates reduced the error of each of the ten foundation