Validating automated eye disease screening AI algorithm in community and in-hospital scenarios

Purpose:To assess the accuracy and robustness of the AI algorithm for detecting referable diabetic retinopathy (RDR), referable macular diseases (RMD), and glaucoma suspect (GCS) from fundus images in community and in-hospital screening scenarios. MethodsWe collected two color fundus image datasets, namely, PUMCH (556 images, 166 subjects, and four camera models) and NSDE (534 images, 134 subjects, and two camera models). The AI algorithm generates the screening report after taking fundus images. The images were labeled as RDR, RMD, GCS, or none of the three by 3 licensed ophthalmologists. The resulting labels were treated as "ground truth" and then were used to compare against the AI screening reports to validate the sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) of the AI algorithm. ResultsOn the PUMCH dataset, regarding the prediction of RDR, the AI algorithm achieved overall results of 0.950 +/- 0.058, 0.963 +/- 0.024, and 0.954 +/- 0.049 on sensitivity, specificity, and AUC, respectively. For RMD, the overall results are 0.919 +/- 0.073, 0.929 +/- 0.039, and 0.974 +/- 0.009. For GCS, the overall results are 0.950 +/- 0.059, 0.946 +/- 0.016, and 0.976 +/- 0.025. ConclusionThe AI algorithm can work robustly with various fundus camera models and achieve high accuracies for detecting RDR, RMD, and GCS.