On the Monocular 3D Pose Estimation for Arbitrary Shaped Needle in Dynamic Scenes: An Efficient Visual Learning and Geometry Modeling Approach

Image-guided needle pose estimation is crucial for robotic autonomous suturing, but it poses significant challenges due to the needle’s slender visual projection and dynamic surgical environments. Current state-of-the-art methods rely on additional prior information (e.g., in-hand grasp, accurate kinematics, etc.) to achieve sub-millimeter accuracy, hindering their applicability in varying surgical scenes. This paper presents a new generic framework for monocular needle pose estimation: Visual learning network for efficient geometric primitives extraction and novel geometry model for accurate pose recovery. To capture needle’s primitives precisely, we introduce a morphology-based mask contour fusion mechanism in a multi-scale manner. We then establish a novel state representation for needle pose and develop a physical projection model to derive its relationship with the primitives. An anti-occlusion objective is formulated to jointly optimize the pose and bias of inference primitives, achieving sub-millimeter accuracy under occlusion scenarios. Our approach requires neither CAD model nor circular shape assumption and can extensively estimate poses of other small planar axisymmetric objects. Experiments on ex-/in-vivo scenarios validate the accuracy of estimated intermediate primitives and final poses of needles. We further deploy our framework on the dVRK platform for automatic and precise needle manipulations, demonstrating the feasibility for use in robotic surgery.