Guidance for Clinical Evaluation under the Medical Device Regulation through Automated Scoping Searches

Featured Application The developed automated search methods can support medical device manufacturers with a first orientation before the clinical evaluation for the European Medical Device Regulation. A first orientation via automated initial scoping searches could lower the cost of scarce resources and manual labor. The Medical Device Regulation (MDR) in Europe aims to improve patient safety by increasing requirements, particularly for the clinical evaluation of medical devices. Before the clinical evaluation is initiated, a first literature review of existing clinical knowledge is necessary to decide how to proceed. However, small and medium-sized enterprises (SMEs) lacking the required expertise and funds may disappear from the market. Automating searches for the first literature review is both possible and necessary to accelerate the process and reduce the required resources. As a contribution to the prevention of the disappearance of SMEs and respective medical devices, we developed and tested two automated search methods with two SMEs, leveraging Medical Subject Headings (MeSH) terms and Bidirectional Encoder Representations from Transformers (BERT). Both methods were tailored to the SMEs and evaluated through a newly developed workflow that incorporated feedback resource-efficiently. Via a second evaluation with the established CLEF 2018 eHealth TAR dataset, the more general suitability of the search methods for retrieving relevant data was tested. In the real-world use case setting, the BERT-based method performed better with an average precision of 73.3%, while in the CLEF 2018 eHealth TAR evaluation, the MeSH-based search method performed better with a recall of 86.4%. Results indicate the potential of automated searches to provide device-specific relevant data from multiple databases while screening fewer documents than in manual literature searches.