Robust Visual Tracking of Robotic Forceps Under a Microscope Using Kinematic Data Fusion

Forceps tracking is an important element of high-level surgical assistance such as visual servoing and motion analysis. This paper describes a robust, efficient tracking algorithm capable of estimating the forceps tip position in an image space by fusing visual tracking data with kinematic information. In visual tracking, the full-state parameters of forceps are estimated using the projective contour models of a 3-D CAD model of the forceps. The likelihood of the contour model is measured using the distance transformation to enable fast calculation, and the particle filter estimates the full state of the forceps. For more robust tracking, the result data obtained from visual tracking are combined with kinematic data that are obtained by forward kinematics and hand-eye transformation. The fusion of visual and kinematic tracking data is performed using an adaptive Kalman filter, and the fused tracking enables the reinitialization of visual tracking parameters when a failure occurs. Experimental results indicate that the proposed method is accurate and robust to image noise, and forceps tracking was successfully carried out even when the forceps was out of view.