A Cross-Modal Deep Learning Method for Enhancing Photovoltaic Power Forecasting with Satellite Imagery and Time Series Data

Accurate photovoltaic (PV) power forecasting improves grid stability and energy utilization efficiency. Integrating large-scale cloud information from satellite imagery can enhance the accuracy of ultra-short-term PV power forecasts. However, existing satellite-based forecasting methods consider the global features of satellite images but overlook the impact of localized cloud movements on future PV generation in the target area. The focus on local information, such as PV time series and nearby clouds in the region of interest, contributes to more efficient feature extraction of satellite images. In this study, a deep learning method is proposed to strengthen the cross-modal correlation of global and local information in satellite image encoding and the multi-modal fusion stage. A novel satellite image encoder is designed by using the dual-branch spatio-temporal vision transformer to compress large-scale cloud features into the features of the region of interest. Satellite image features are then combined with PV time-series features using across transformer with rotary position embedding. The proposed method was validated using data from ten PV stations, demonstrating forecast skill of 47.29%-58.23% for PV power forecasts up to 4 h ahead. Compared to ViT, ViViT, CrossViT, and Perceiver, the proposed method achieves an average improvement of 2.39%-3.75%, and a minimum of 8.98% improvement in scenarios where PV time-series data is unavailable. Moreover, the proposed method outperforms the state-ofthe-art methods by 2.85%-5.53%. The experimental results highlight that the proposed method shows accurate and robust forecasting performance and is a reliable alternative to PV power forecasting.