3D hand pose estimation and reconstruction based on multi-feature fusion

Jiye Wang,Xuezhi Xiang, Shuai Ding,Abdulmotaleb El Saddik

Journal of Visual Communication and Image Representation(2024)

引用 0|浏览2
3D hand pose estimation and shape reconstruction is to recover the hand joint points and hand mesh vertices coordinates from the image. However, existing methods usually use only the high-level semantic features extracted by the backbone network to represent the hand mesh vertex features, which leads to a single representation of the hand vertices features and cannot fully utilize the feature information extracted by the network. In this paper, we propose a method for real-time 3D reconstruction of hands from a single RGB image, which enriches the 3D semantic information of the mesh vertices through multi-feature fusion. Firstly, we regress the 2D features of mesh vertices through Integral Pose Regression (IPR) and regard them as prior information to 3D features. Then we design a Multi-Scale Sampling(MSS) module to extract multi-scale information. Finally we fuse 2D prior features, multi-scale features, and high-level semantic features extracted by backbone to represent 3D initial feature. Additionally, we propose an Multi-Root(MR) loss function to address the imbalance problem caused by a single root joint. The experimental results indicate that our network achieves competitive performance on the FreiHAND and HO-3D public dataset, achieving fast inference speed with fewer parameters.
Hand pose estimation,Shape reconstruction,Multi-feature fusion,Multi-scale,Multi-root loss
AI 理解论文
Chat Paper