Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology

The medical imaging community has embraced Machine Learning (ML) as evidenced by the rapid increase in the number of ML models being developed, but validating and deploying these models in the clinic remains a challenge. The engineering involved in integrating and assessing the efficacy of ML models within the clinical workflow is complex. This paper presents a general-purpose, end-to-end, clinically integrated ML model deployment and validation system implemented at UCSF. Engineering and usability challenges and results from 3 use cases are presented. A generalized validation system based on free, open-source software was implemented, connecting clinical imaging modalities, the Picture Archiving and Communication System (PACS), and an ML inference server. ML pipelines were implemented in NVIDIA’s Clara Deploy framework with results and clinician feedback stored in a customized XNAT instance, linked within PACS. Prospective clinical validation studies of 3 ML models were conducted, with data routed from multiple clinical imaging modalities and PACS. Completed validation studies provided expert clinical feedback on model performance and usability, plus system reliability and performance metrics. Clinical validation of ML models entails assessing model performance, impact on clinical infrastructure, robustness, and usability. Study results must be easily accessible to participating clinicians but remain outside the clinical record. Building a system that generalizes and scales across multiple ML models takes the concerted effort of software engineers, clinicians, data scientists, and system administrators, and benefits from the use of modular open-source software. The present work provides a template for institutions looking to translate and clinically validate ML models in the clinic, together with required resources and expected challenges. Author summary Academic medical centers gather and store vast quantities of digital data, and with the increase in accessibility of Machine Learning (ML) techniques, there has been an explosion of ML model development in the medical imaging community. Most of this work remains in research, though, and connecting ML models to the clinic for testing on live patient data and integration into the clinical workflow remains a challenge and impedes clinical impact. We present a general-purpose system, implemented and deployed at UCSF, for in-clinic validation of ML models and their incorporation into patient care. This work, based on free and open-source software packages, can serve as a template for other institutions looking to solve ML’s “last mile” problem and move their models out of research and into the clinic. ### Competing Interest Statement NVIDIA provided 4 T4 cards as a grant to UCSF. ### Funding Statement This work had no direct funding sources. ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Not Applicable The details of the IRB/oversight body that provided approval or exemption for the research described are given below: Validation studies received UCSF institutional review board approvals with consent waivers. I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Not Applicable I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Not Applicable I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Not Applicable All relevant data are within the manuscript.