Can Ensemble Machine Learning Improve the Accuracy of Severe Maternal Morbidity Screening in a Perinatal Database?

Background: Severe maternal morbidity (SMM) is an important maternal health indicator, but existing tools to identify SMM have substantial limitations. Our objective was to retrospectively identify true SMM status using ensemble machine learning in a hospital data-base and to compare machine learning algorithm performance with existing tools for SMM identification. Methods: We screened all deliveries occurring at Magee-Womens Hospital, Pittsburgh, PA (2010-2011 and 2013-2017) using the Centers for Disease Control and Prevention list of diagnoses and procedures for SMM, intensive care unit admission, and/or prolonged postpartum length of stay. We performed a detailed medical record review to confirm case status. We trained ensemble machine learning (SuperLearner) algorithms, which "stack" predictions from multiple algorithms to obtain optimal predictions, on 171 SMM cases and 506 non-cases from 2010 to 2011, then evaluated the performance of these algorithms on 160 SMM cases and 337 non-cases from 2013 to 2017. Results: Some SuperLearner algorithms performed better than existing screening criteria in terms of positive predictive value (0.77 vs. 0.64, respectively) and balanced accuracy (0.99 vs. 0.86, respectively). However, they did not perform as well as the screening criteria in terms of true-positive detection rate (0.008 vs. 0.32, respectively) and performed similarly in terms of negative predictive value. The most important predictor variables were intensive care unit admission and prolonged postpartum length of stay. Conclusions: Ensemble machine learning did not globally improve the ascertainment of true SMM cases. Our results suggest that accurate identification of SMM likely will remain a challenge in the absence of a universal definition of SMM or national obstetric surveillance systems.