Towards in-utero Navigational Assistance: A Multi Task Neural Network for Segmentation and Pose Estimation in Fetoscopy

Fetoscopic laser coagulation (FLC) is the most prevalent therapy for treating twin-to-twin transfusion syndrome (TTTS). A rigid or flexible fetoscope is inserted into the uterine cavity through a small incision in this minimally invasive technique. The procedure consists of surveying the placental surface, identifying anastomosing vessels, and coagulation. This paper presents a multi-task neural network model to segment the vasculature from the fetoscopic images and estimate the relative orientation and distance of the placental surface to assist the surgeons. This work also presents a method to use virtual reality (VR) to generate datasets for training and testing. A silicon-based placenta phantom was created in a planar form of 29 × 29 cm with realistic vasculature. A scanned image of this phantom was manually segmented as the ground truth. Both the color image of the placenta and segmented ground truth were placed in the VR simulator. The virtual camera is moved by pre-defined camera motions, which then renders fetoscopic placenta images and their corresponding segmented ground truth without the need for manual segmentation. The network achieved a dice coefficient of 0.8 on the segmentation task and 87% accuracy on the regression task. The network's capacity to identify vessels was also evaluated using actual images from a flexible fetoscope's chip-on-tip camera.