An Evaluation of Methods for Monitoring Annual Quality Measures by Month to Predict Year-End Values

Abstract Background An increasing number of healthcare quality measures are designed for annual reporting. These measures require an entire year of data to accurately report the percentage of patients who met the measure. Annual measures give providers latitude to prioritize clinical workload and patient needs; however, they do not provide a direct means to monitor performance throughout the reporting year. Although there are many possible methods for measuring annual measures at finer-grained timescales, our applied work showed that the most obvious methods could give a misleading and inaccurate view of progress throughout the year. Neither the definitions of the annual measures, nor the research literature, provided any guidance on the best methods for interim monitoring of annual measures. Objective Our objective was to evaluate four different methods for monitoring annually reported quality measures monthly to best predict year-end performance throughout the reporting year. Methods We developed four methods for monitoring annual measures by month: 1) Monthly Proportion: The proportion of patients with one or more encounters in the month who still needed to meet the measure at their first encounter of the month and met the measure by the end of the month; (2) Monthly Lookback Proportion: The proportion of patients seen in the month who met the measure by the end of the month, regardless of whether it was met in that month or previously in the reporting year; (3) Rolling 12 Month: The annual measure reported as if each month was the twelfth month of a twelve-month reporting period; and (4) YTD (Year-to-Date) Cumulative: The proportion of patients with one or more visits from the start of the reporting year through the month who satisfy the measure. We applied each method to two annual dental quality measures using data from two reporting years, and four different dental sites. We used mean squared error (MSE) to evaluate year-end predictive performance. Results Method 3 (Rolling 12 Month) had the lowest MSE in 11 out of 16 cases (2 measures X 2 years X 4 sites) and lowest total MSE (262.39) across all 16 cases. In 5 of the 16 cases, YTD Cumulative had the lowest MSE. Conclusions The Rolling 12 Month method was best for predicting the year-end value across both measures and all four sites.