A Framework for Evaluating Geomagnetic Indices Forecasting Models

The use of Deep Learning models to forecast geomagnetic storms is achieving great results. However, the evaluation of these models is mainly supported on generic regression metrics (such as the Root Mean Squared Error or the Coefficient of Determination), which are not able to properly capture the specific particularities of geomagnetic storms forecasting. Particularly, they do not provide insights during the high activity periods. To overcome this issue, we introduce the Binned Forecasting Error to provide a more accurate assessment of models' performance across the different intensity levels of a geomagnetic storm. This metric facilitates a robust comparison of different forecasting models, presenting a true representation of a model's predictive capabilities while being resilient to different storms duration. In this direction, for enabling fair comparison among models, it is important to standardize the sets of geomagnetic storms for model training, validation and testing. To do this, we have started from the current sets used in the literature for forecasting the SYM-H, enriching them with newer storms not considered previously, focusing not only on disturbances caused by Coronal Mass Ejections but also addressing High-Speed Streams. To operationalize the evaluation framework, a comparative study is conducted between a baseline neural network model and a persistence model, showcasing the effectiveness of the new metric in evaluating forecasting performance during intense geomagnetic storms. Finally, we propose the use of preliminary measurements from ACE to evaluate the model performance in settings closer to an operational real-time scenario, where the forecasting models are expected to operate. Using Machine Learning (ML) for forecasting geomagnetic indices has seen a notable increase in popularity, especially with the constantly increasing available data from space probes and ground-based magnetometers. One critical aspect that defines the success of ML models in this domain is the quality and representativeness of the data. The evaluation of previous models in the field often relied in conventional metrics that did not give a complete evaluation of the particularities of the geomagnetic storms. To address this, we introduce the Binned Forecasting Error, a metric designed to provide a more rigorous and complete evaluation of model performance, taking into account the particular characteristics of geomagnetic storms and the challenges inherent in predicting them. Considering that, we have reviewed and expanded the selected storms for the SYM-H index to include a broader spectrum of geomagnetic events. Plus, we suggest using preliminary data from ACE to test these models in more realistic scenarios, similar to how they would be used in an operational scenario. A common framework for evaluating geomagnetic index forecasting models improves the quality of research in a growing field We introduce the Binned Forecasting Error for better capturing the relevance of large geomagnetic storms in forecasting models We expand current assessed geomagnetic storms sets to improve training and testing of geomagnetic forecasting models