Virtual vs. Reality: External Validation of COVID-19 Classifiers using XCAT Phantoms for Chest Radiography

Many published studies use deep learning models to predict COVID-19 from chest x-ray (CXR) images, often reporting high performances. However, the models do not generalize well on independent external testing. Common limitations include the lack of medical imaging data and disease labels, leading to training on small datasets or drawing classes from different institutions. To address these concerns, we de signed an external validation study of deep learning classifiers for COVID-19 in CXR images including XCAT phantoms as well. We hypothesize that a simulated CXR image dataset obtained from the XCAT phantom allows for better control of the dataset including pixel-level ground truth. This setup allows for multiple advantages: First, we can validate the publicly available models using simulated chest x-rays. Secondly, we can also address clinically relevant questions with this setup such as effect of dose levels and size of COVID-19 pneumonia in performance of deep learning classifier. We design a study to address these concerns on this work. For the first part, we validate the performance of a publicly available model from the University of Waterloo trained on a large clinical dataset by testing on another large BIMCV-RSNA clinical dataset as well as simulated XCAT CXRs. We also trained an in-house classification model on the same BIMCV-RSNA dataset and validate the model on both an internal test set (AUC 0.71) and the simulated set CVIT-COVID (AUC 0.69) and COVIDx CXR-2 (Waterloo) (AUC 0.62). For the second part, we further investigate the role of clinically relevant factors such as dose levels in CXR and size of COVID-19 pneumonia in performance of deep learning-based classifiers using XCAT phantoms. As the virtual imaging trial approach allows generating images of varied dose and COVID-19 pneumonia sizes, our setup allows us to investigate the role of different features. The results show promise of using the virtual image trial approach to validate deep learning models and better control of study design to address clinically significant factors.