Deep learning-based solar image captioning

Solar images are essential for identifying and predicting solar phenomena, and have been used as key information for analyzing space weather. In this paper, we propose a solar image captioning method that applies a transformer-based deep learning (DL) natural language processing method. In addition, we provide a new DeepSDO description dataset for training solar image captioning models. First, we develop the DeepSDO description dataset using solar image data from Korean Data Center for solar dynamics observatory (SDO) and scripts from the National Aeronautics and Space Administration (NASA) SDO gallery website. The DeepSDO description dataset includes nine solar events: sunspots, flares, prominences, prominent eruptions, coronal holes, coronal loops, filaments, active regions, and eclipses. Second, we train the DL-based image captioning model, the meshed-memory transformer, using the DeepSDO description dataset. The experimental results show that the proposed method outperforms other benchmark methods in terms of four evaluation metrics. This study demonstrates that DL-based image captioning can successfully generate solar image captions for multiple solar features, and could potentially be used in other themes of solar physics and space weather. (c) 2024 COSPAR. Published by Elsevier B.V. All rights reserved.