Inspecting the quality of care: a comparison of CUSUM methods for inter hospital performance

During the past 14 years, a clinical audit has been used in the Netherlands to provide hospitals with data on their performance in colorectal cancer care. Continuous feedback on the quality of care provided at each hospital is essential to improve patient outcomes. It is unclear which methods should be used to generate most informative output for the identification of potential quality issues. Our aim is to compare the commonly employed funnel plot with existing cumulative sum (CUSUM) methodology for the evaluation of postoperative survival and hospital stay outcomes of patients who underwent colorectal surgery in the Netherlands. Data from the Dutch ColoRectal Audit on 25367 patients in the Netherlands who underwent surgical resection for colorectal cancer in 71 hospitals between 2019 and 2021 is used to compare four methods for the detection of deviations in the quality of care. Two methods based on binary outcomes (funnel plot, binary CUSUM) and two CUSUM charts based on survival outcomes (BK-CUSUM and CGR-CUSUM) are considered. A novel approach for determining hospital specific control limits for CUSUM charts is proposed. The ability to detect deviations as well as the time until detection are compared for the four methods. Charts were constructed for the inspection of both postoperative survival and hospital stay. Methods using survival outcomes always yielded faster detection times compared to approaches employing binary outcomes. Detections between methods mostly coincided for postoperative survival. For hospital stay detections varied strongly, with methods based on survival outcomes signalling over half the hospitals. Further pros and cons as well as pitfalls of all methods under consideration are discussed. Methodology for the continuous inspection of the quality of care should be tailored to the specific outcome. Properly understanding how the mechanism of a control chart functions is crucial for the correct interpretation of results. This is particularly true for CUSUM charts, which require the choice of a parameter that greatly influences the results. When applying CUSUM charts, consideration of these issues is strongly recommended.