Automatic Biopsy Tool Presence and Episode Recognition in Robotic Bronchoscopy Using a Multi-Task Vision Transformer Network

Automatic recognition of surgical workflow is a growing area of interest with significant potential to become part of context-aware decision-support systems in future enhanced ORs and clinical suites. Applications range from post-operative analysis to intra-operative monitoring to providing automated assistance to the clinical staff. This work proposes, for the first time, automatic tool presence and episodes recognition in bronchoscopy images, using a newly annotated video dataset obtained from robotic bronchoscopy procedures. A novel multi-task architecture that utilizes Vision Transformers for the episode recognition task is used to achieve improved accuracy while reducing the computational burden during the training stage. The method is thoroughly validated on clinical procedure video data obtained across 135 procedures, displaying improved performance and training times compared to various state-of-the-art methods.