Augmented Reality Visualization and Interaction for COVID-19 CT-Scan NN Automated Segmentation: A Validation Study

Although medical imaging technology has persisted in evolving over the last decades, the techniques and technologies used for analytical and visualization purposes have remained constant. Manual or semiautomatic segmentation is, in many cases, complicated. It requires the intervention of a specialist and is time-consuming, especially during the coronavirus disease (COVID-19) pandemic, which has had devastating medical and economic consequences. Processing and visualizing medical images with advanced techniques represent medical professionals’ breakthroughs. This article studies how augmented reality (AR) and artificial intelligence (AI) can transform medical practice during COVID-19 and post-COVID-19 pandemic. Here, we report an AR visualization and interaction platform; it covers the whole process from uploading chest computed tomography (CT)-scan images to automatic segmentation-based deep learning, 3-D reconstruction, 3-D visualization, and manipulation. AR provides a more realistic 3-D visualization system, allowing doctors to effectively interact with the generated 3-D model of segmented lungs and COVID-19 lesions. We use the U-Net neural network (NN) for automated segmentation. The statistical measures obtained using the Dice score, pixel accuracy, sensitivity, ${G}$ -mean, and specificity are 0.749, 0.949, 0.956, 0.955, and 0.954, respectively. The user-friendliness and usability are objectified by a formal user study that compared our AR-assisted design to the standard diagnosis setup. One hundred and six doctors and medical students, including eight senior medical lecturers, volunteered to assess our platform. The platform could be used as an aid-diagnosis tool to identify and analyze the COVID-19 infectious or as a training tool for residents and medical students. The prototype can be extended to other pulmonary pathologies.