Effect of an artificial intelligence tool on management decisions for indeterminate pulmonary nodules

Artificial intelligence (AI) radiomics-based tools demonstrate promise for indeterminate pulmonary nodule (PN) malignancy risk stratification.1 We performed a secondary analysis of a previous multi-reader, multi-case study2 to evaluate the effect of an AI tool on clinicians' PN management decisions. The details of this study have been previously described.2 Briefly, 12 readers (6 radiologists, 6 pulmonologists) independently evaluated 300 indeterminate PN cases using solely axial CT chest scan imaging data. PNs were 5–30 mm in maximal diameter, and 50% were malignant. The AI tool assessed was the Lung Cancer Prediction Convolutional Neural Network (Virtual Nodule Clinic, version 2.0.0; Optellum Ltd, Oxford, UK).3, 4 This tool calculates a Lung Cancer Prediction (LCP) score describing PN malignancy risk on a decile scale from 1 to 10 assuming a malignancy prevalence of 30%. For each case, each reader independently provided estimates of malignancy risk (0%–100%) and management decision (no follow-up, ≥6-month CT follow-up, 6-week to 6-month CT follow-up, immediate imaging follow-up, non-surgical biopsy, or surgical resection) before and after being shown the LCP score. We defined appropriate management of malignant PNs as non-surgical biopsy and surgical resection. For benign PNs, no follow-up or imaging follow-up were deemed appropriate. We classified immediate imaging as appropriate management for all PNs. The median LCP score for malignant PNs was 9 (IQR, 8–10) and 5 (IQR, 2–7) for benign PNs (p < 0.001). Among malignant PNs, the average reader malignancy risk estimate was 60.2% (SD, 31.7%) without the AI tool compared to 69.0% (SD, 28.6%) with it (p < 0.001). Among benign PNs, the average reader malignancy risk estimate was 23.4% (SD, 28.1%) without the AI tool compared to 21.0% (SD, 26.9%) with it (p = 0.01). The distributions of management decisions are displayed in Figure 1. Overall, the proportion of cases with appropriate management decisions increased from 79.5% (SD, 5.7%) to 84.1% (SD, 6.6%) with AI (p = 0.008). Among malignant PNs, on average readers selected immediate imaging, biopsy, or surgical resection in 71.9% (SD, 14.0%) of cases without use of AI compared to 81.4% (SD, 13.7%) with the AI tool (p < 0.001). Among benign PNs, on average readers selected no action, short-term, long-term, or immediate follow-up imaging in 87.2% (SD, 10.4%) of cases without and 88.7% (SD, 11.1%) with the AI tool, respectively (p = 0.19). We found that use of an AI tool was associated with an increase of the average proportion of cases with appropriate management decisions from 79.5% to 84.1%. This was largely driven by a 10 percentage point increase in malignant PNs appropriately managed with immediate imaging or tissue sampling. On the other hand, we did not observe a statistically significant difference in the management of benign PNs with use of the AI tool. Taken together, these results suggest that the previously demonstrated improvement in diagnostic accuracy with use of an AI tool may translate into meaningful changes in clinical management decisions and promote earlier diagnostic evaluation of malignant PNs, which may ultimately lead to increased timeliness of appropriate clinical treatment for thoracic malignancies. Roger Kim: Conceptualization (lead); formal analysis (lead); methodology (equal); visualization (lead); writing – original draft (lead); writing – review and editing (lead). Jason L. Oke: Formal analysis (supporting); methodology (supporting); writing – review and editing (equal). Travis L. Dotson: Data curation (equal); writing – review and editing (equal). Christina Bellinger: Data curation (equal); writing – review and editing (equal). Anil Vachani: Conceptualization (equal); data curation (equal); formal analysis (supporting); methodology (supporting); supervision (lead); writing – original draft (supporting); writing – review and editing (equal). Research funding: Roger Y. Kim was supported by the National Cancer Institute of the National Institutes of Health (Award Number 5UM1CA221939). Jason L. Oke was part-funded by the NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust. This study was funded by Optellum Ltd. Roger Y. Kim reports research funding from Siemens outside of the submitted work. Anil Vachani reports research funding from MagArray, Inc., Broncus Medical, and PreCyte, Inc. and a consulting role with Novocure and Johnson & Johnson, outside of the submitted work. The remaining authors reported no relevant conflicts of interest. The use of deidentified imaging studies complied with Health Insurance Portability and Accountability Act guidelines, and the need for informed consent was waived by local institutional review boards.