Capturing fine-grained details for video-based automation of suturing skills assessment

Objectives Manually-collected suturing technical skill scores are strong predictors of continence recovery after robotic radical prostatectomy. Herein, we automate suturing technical skill scoring through computer vision (CV) methods as a scalable method to provide feedback. Methods Twenty-two surgeons completed a suturing exercise three times on the Mimic™ Flex VR simulator. Instrument kinematic data (XYZ coordinates of each instrument and pose) were captured at 30 Hz. After standardized training, three human raters manually video segmented suturing task into four sub-stitch phases ( Needle handling, Needle targeting, Needle driving, Needle withdrawal ) and labeled the corresponding technical skill domains ( Needle positioning, Needle entry, Needle driving, and Needle withdrawal) . The CV framework extracted RGB features and optical flow frames using a pre-trained AlexNet. Additional CV strategies including auxiliary supervision (using kinematic data during training only) and attention mechanisms were implemented to improve performance. Results This study included data from 15 expert surgeons (median caseload 300 [IQR 165–750]) and 7 training surgeons (0 [IQR 0–8]). In all, 226 virtual sutures were captured. Automated assessments for Needle positioning performed best with the simplest approach (1 s video; AUC 0.749). Remaining skill domains exhibited improvements with the implementation of auxiliary supervision and attention mechanisms when deployed separately (AUC 0.604–0.794). All techniques combined produced the best performance, particularly for Needle driving and Needle withdrawal (AUC 0.959 and 0.879, respectively). Conclusions This study demonstrated the best performance of automated suturing technical skills assessment to date using advanced CV techniques. Future work will determine if a “human in the loop” is necessary to verify surgeon evaluations.