PMMWPoint: Weakly Supervised Dense Keypoint Detection and Description for Passive Millimeter-Wave Image Matching

Passive millimeter-wave (PMMW) imager has become a common device for detecting concealed objects in security screening. However, the current detection methods rely primarily on 2-D information, which is inefficient for objects with similar grayscale values to the human body. Exploiting the 3-D features of the objects will be conducive to tackling this challenge, where image matching is generally the first step before performing the 3-D reconstruction. However, the existing matching methods struggle to produce reliable keypoints in textureless and noisy PMMW images, since they mainly focus on texture regions with distinct features, such as corners and lines. To tackle this challenge, we present a weakly supervised framework to learn keypoints in PMMW images, named as PMMWPoint. Specifically, we first pretrain a diffusion model on unlabeled PMMW images in an unsupervised manner for better modeling the intricate characteristics of PMMW images. Then we employ a few labeled keypoints as guidance for network to detect keypoints from textureless regions. To compensate for the sparsity of manual labeling, a self-paced keypoint augmentation (SPKA) strategy is introduced to progressively increase the quantity and quality of ground-truth labels during training. In addition, we propose an improved contrastive loss for better descriptor learning by integrating the information from detection and description branches. Extensive experiments prove the superiority of PMMWPoint in producing dense and accurate keypoints, which deliver a notable improvement of +4.8% and +1.2% in homography accuracy over unsupervised baselines for PMMW and Hpatches datasets, respectively. Through our investigation, our work represents the first effort to address the problem of PMMW image matching.