LeGFusion: Locally-enhanced Global Learning for Multi-Modal Image Fusion

Multi-modal image fusion (MMIF) can provide more comprehensive scene characteristics by synthesizing a single image from multi-sensor images of the same scene, which works out the limitation of single-type hardwares. To handle MMIF tasks, current deep learning-based methods usually employ convolutional neural networks or combine transformer to extract local and global contextual information of source images. However, none of existing works fully explore contextual information both across modalities and within single modalities, leading to limited fusion results. To this end, we propose a new MMIF method via locally-enhanced global learning, termed as LeGFusion. Specifically, the network of LeGFusion is devised based on locally-enhanced transformer block (LETB), which can capture long-range dependencies benefiting from non-overlapping window-based self-attention while capturing useful local context with the utilization of the convolution operator into transformer. On the one hand, several LETBs are deployed to extract global contexts from each modality while emphasizing its local information. On the other hand, the fusion module that also consists of LETBs is designed to integrate multi-modal features by perceiving cross-modal local and global interactions. Powered by these intra-modal and inter-modal contextual information exploration, the proposed LeGFusion enjoys a high capability in capturing significant complementary information for image fusion. Extensive experiments are conducted on two types of MMIF tasks, including infrared-visible image fusion and medical image fusion. The qualitative and quantitative evaluation results demonstrate the superiority of our LeGFusion over state-of-the-art methods. Furthermore, we validate the generalization ability of LeGFusion without fine-tuning, and achieve fantastic results.