QRCODE: Quasi-residual Convex Deep Network for Fusing Misaligned Hyperspectral and Multispectral Images

Considering that hyperspectral image (HSI) is often of lower spatial resolution when compared to multispectral image (MSI), an economical approach for obtaining a high-spatial-resolution (HSR) HSI is to fuse the acquired HSI and MSI, thereby greatly facilitating the subsequent material identification and classification in satellite remote sensing. As satellite-acquired HSI and MSI are often misaligned, the proposed deep neural network does not require the input HSI/MSI to be spatially co-registered, making the challenging fusion network design even more difficult. In this study, we propose a streamlined and efficient convex model integrated into the sub-network, which obviates the need for complex network structures in learning spatial-spectral relationships, effectively guiding the quasi-residual learning task in our alignment-free fusion network. The convex sub-network is a low-rank model that leverages the convex geometric structure implicitly embedded in the hyperspectral signature space. To address the misalignment between HSI and MSI effectively, we introduce a novel Shifted Window Attention Module (SWAM) that exploits the neighboring correlation in the feature domain, significantly enhancing the performance and stability of the fusion task. Capitalizing on the redundancy among spectrums, we employ grouped convolution to decrease the computational complexity without causing additional performance degradation. The proposed Quasi-residual Convex Deep Network (QRCODE) demonstrates state-of-the-art performance in alignment-free HSI/MSI fusion tasks.