Pos0377 accuracy of an ai-based symptom checker and an online self-referral tool in rheumatology: results from a multicenter randomized controlled trial

Background Inflammatory rheumatic diseases (IRD) are often diagnosed too late due to non-specific symptoms and the lack of specialists in rheumatology. Digital diagnostic decision support systems (DDSS) promise to accelerate diagnosis and decrease the overall healthcare burden. Objectives To assess the ability of an artificial intelligence (AI)-based symptom checker (Ada) and an online self-referral tool (Rheport) to diagnose inflammatory rheumatic diseases (IRD). Methods In a prospective, multicenter open-label controlled randomized crossover trial patients newly presenting to a rheumatology center were randomly assigned in a 1:1 ratio to complete a symptom assessment with Ada or Rheport followed by a crossover to the other respective diagnostic decision support system (DDSS). The primary outcome was correct identification of a patient with IRD by the DDSS, defined as the presence of any IRD in the list of suggested diagnoses with Ada or a pre-specified threshold score with Rheport. Physicians’ diagnosis was the gold standard. Results In total, 600 patients were included and 214 (36%) patients were eventually diagnosed with an IRD by a physician. Rheport showed a sensitivity of 62% and specificity of 47% for IRDs. Ada’s top 1 (D1) and top 5 disease suggestions (D5) showed a sensitivity of 52% and 66% and a specificity of 68% and 54% concerning IRDs, respectively. Ada, in comparison to Rheport, was more likely to correctly identify patients with an IRD when used as the first DDSS (OR: 1.09, 95% CI: 1.01 to 1.18) however this finding was not consistent after cross-over (OR: 0.97, 95% CI: 0.90 to 1.05). Conclusion The diagnostic capability of both DDSS for IRDs was not promising in this high-prevalence patient population referred for subspecialty evaluation. Although the overall numbers suggest that AI-based Ada demonstrated a slightly higher specificity and sensitivity compared to the questionnaire-based Rheport, Ada was not consistently better than Rheport in correctly identifying patients with an IRD when the use sequence of the apps was taken into account. Our results indicate that, strict regulation and drastic improvement is necessary to ensure safety and effectiveness of DDSS. Acknowledgements This study was partially funded by Novartis Pharma GmbH. Disclosure of Interests Johannes Knitza Speakers bureau: Abbvie, Novartis, Lilly, Medac, BMS, Sanofi, Amgen, Gilead, UCB, ABATON, GSK, Werfen, Vila Health, Böhringer Ingelheim, Janssen, Galapagos, Chugai, Celltrion, Grant/research support from: This study has been partially supported by Novartis Pharma GmbH. Others: Abbvie, Novartis, Thermo Fisher, UCB, ABATON, Sanofi, DFG, EIT Health, Koray Tascilar: None declared, Franziska Fuchs: None declared, Jacob Mohn: None declared, David Simon: None declared, Arnd Kleyer: None declared, Christina Bergmann: None declared, Hannah Labinsky: None declared, Harriet Morf: None declared, Elizabeth Araujo: None declared, Daniela Bohr: None declared, Felix Muehlensiepen: None declared, Matthias Englbrecht: None declared, Wolfgang Vorbrüggen: None declared, Cay-Benedict von der Decken: None declared, Stefan Kleinert: None declared, Andreas Ramming: None declared, Joerg Distler: None declared, Peter Bartz-Bazzanella: None declared, Nicolas Vuillerme: None declared, Georg Schett: None declared, Martin Welcker Grant/research support from: Novartis Pharma GmbH, Axel Hueber Grant/research support from: Novartis Pharma GmbH.