Clinical simulation in the regulation of software as a medical device (SaMD): an eDelphi study

Abstract Accelerated digitalization in the health sector requires the development of appropriate evaluation methods to ensure digital health technologies (DHTs) are safe and effective. Clinical simulation can be used by researchers to test DHTs in an agile and low-cost way, yet there is limited research on criteria to assess the robustness of simulations and subsequently, their relevance for a regulatory decision. The aim of this study was to gain consensus the research question “What criteria should be used to assess clinical simulation being used to generate evidence for software as a medical device (SaMD)?” 39 international experts in the digital health field, including academics, regulators, policy makers, and industry representatives, participated in a three-round eDelphi exercise. Options were generated through the scoping questionnaire around key themes identified from the literature to obtain a comprehensive list of criteria and voted upon in two subsequent questionnaire rounds. Consensus was defined by two criteria: if <10% of the panelists deemed the criteria as ‘not important’ or ‘not important at all’ and >60% ‘important’ or ‘very important’. 43 criteria gained consensus from the panelists across seven domains and the Simulation for Regulation of SaMD (SIROS) framework was developed. We highlight key areas of concern identified by panelists, specifically on the importance of fidelity of simulation and its reporting, and the challenge of bias. Future research should prioritise the development of safe and effective SaMD, while implementing the criteria generated for regulating DHT based on clinical simulation evidence can enable faster uptake of technologies.