Enhancing multi-scale information exchange and feature fusion for human pose estimation

Multi-scale feature fusion is an important part of modern network architectures to extract more comprehensive information for most computer vision tasks, such as semantic segmentation and keypoint estimation. However, most existing multi-scale methods add fusion connections between layers or branches directly, which inevitably ignores the semantic information discrepancy between feature maps with different resolutions and depths. Moreover, inappropriate fusion connections may lead to the loss of channel-wise and spatial information. In this paper, we propose a method to enhance and refine multi-scale feature fusion for human pose estimation by employing two attention mechanisms. Specifically, we present a novel multi-head spatial attention (MHSA), which is employed to model context information of the intermediate feature maps and reinforce important local features. Meanwhile, we utilize the position channel attention (PCA) to capture long-range dependencies while retaining the important position information in the attention maps. Combining with the modules of MHSA and PCA, we design an enhanced multi-scale feature fusion network (EMF-HRNet) based on the high-resolution network (HRNet). Our proposed EMF-HRNet yields better results with repeated multi-scale information exchange and feature fusion units. Extensive experiments on two common benchmarks, COCO Keypoint dataset and MPII Human Pose dataset, show that our method significantly improves the performance of state-of-the-art pose estimation methods.