Evaluation of a digital system to predict unplanned admissions to the intensive care unit: A mixed-methods approach

Background: We have developed the Hospital Alerting Via Electronic Noticeboard (HAVEN) which aims to identify hospitalised patients most at risk of reversible deterioration. HAVEN combines patients' vital-sign measurements with laboratory results, demographics and comorbidities using a machine learnt algorithm. Objectives: The aim of this study was to identify variables or concepts that could improve HAVEN predictive performance. Methods: This was an embedded, mixed methods study. Eligible patients with the five highest HAVEN scores in the hospital (i.e., 'HAVEN Top 5') had their medical identification details recorded. We conducted a structured medical note review on these patients 48 hours post their identifiers being recorded. Methods of constant comparison were used during data collection and to analyse patient data. Results: The 129 patients not admitted to ICU then underwent constant comparison review, which produced three main groups. Group 1 were patients referred to specialist services (n = 37). Group 2 responded to ward-based treatment, (n = 38). Group 3 were frail and had documented treatment limitations (n = 47). Conclusions: Digital-only validation methods code the cohort not admitted to ICU as 'falsely positive' in sensitivity analyses however this approach limits the evaluation of model performance. Our study suggested that coding for patients referred to other specialist teams, those with treatment limitations in place, along with those who are deteriorating but then respond to ward-based therapies, would give a more accurate measure of the value of the scores, especially in relation to cost-eectiveness of resource utilisation.