MIMO-SST: Multi-Input Multi-Output Spatial-Spectral Transformer for Hyperspectral and Multispectral Image Fusion

The current advanced hyperspectral super-resolution methods utilize convolutional neural networks (CNNs) that are either deeper or wider. These networks are designed to acquire end-to-end mapping capability, facilitating the transformation from low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs) to high-resolution HSIs (HR-HSIs). The existing methods lack the capability to capture details and structures in the image effectively, while multi-input multi-output methods can address this issue efficiently. Therefore, this article proposes a novel network architecture named multi-input multi-output spatial-spectral transformer (MIMO-SST). To apply the multi-input multi-output methods in HSI fusion, specifically integrating the spatial-spectral information of LR-HSI and HR-MSI, we introduce multihead feature map attention, multihead feature channel attention, and a multiscale convolutional gated feedforward network, constructing the proposed mixture spatial-spectral Transformer. Moreover, to enhance the expressive power of image edges and recover the sharpened structure details, this study incorporates a novel wavelet-based high-frequency loss into the ultimate comprehensive loss, with the objective of refining the reconstruction of high-frequency details. Experimental studies on three simulated datasets and one real-world dataset demonstrate that the proposed method in this study outperforms contemporary state-of-the-art methods in terms of performance. It is noteworthy that our method exhibits a 0.85-dB improvement in terms of the peak signal-to-noise ratio (PSNR) metric on the Columbia computer vision laboratory (CAVE) dataset compared to state-of-the-art methods. Our code is publicly available at https://github.com/Freelancefangjian/MIMO-SST.