Wisdom of the Experts versus Wisdom of the Crowd in Hospital Quality Ratings: Cross-Sectional Analyses of Google Ratings (Preprint)

BACKGROUND Popular online portals provide free and convenient access to user-generated quality reviews. Centers for Medicare and Medicaid Services (CMS) also provide patients with Hospital Compare Star Ratings (HCSR), a single public measure of hospital quality aggregating multiple quality dimensions. Consumers often use crowdsourced hospital ratings on platforms such as Google to select hospitals, but it is unknown if these ratings reflect a comprehensive measure of clinical quality. OBJECTIVE We analyze if Google online quality ratings, which reflect the wisdom of the crowd, are associated with HCSR, which reflect the wisdom of the experts. CMS revised the methodology of assigning star ratings to hospitals. Therefore, we analyze these associations before and after the 2021 revisions of the CMS rating system. METHODS We extracted Google ratings using Application Programming Interface (API) in June 2020. The HCSR data of April 2020 (before the revision of HCSR methodology) and April 2021 (after the revision of HCSR methodology) were obtained from CMS’ Hospital Compare (HC) website. We also extracted scores for the individual components of hospital quality for each of the hospitals in our sample using the code provided by HC. Fractional Response Model (FRM) was used to estimate the association between Google Ratings and HCSR and individual components of quality. RESULTS Results indicate that Google ratings are statistically associated with HCSR (P<.001) after controlling for hospital level effects. A one star improvement in CMS ratings before the change in methodology (after the change in methodology) is expected to increase the Google ratings by 0.145 (0.135) on average (95% CI 0.127- 0.163; P<.001, 95% CI 0.116-0.153; P<.001). The analyses with individual components of hospital quality reveal that Google ratings are not associated with components of HCSR that require medical expertise such as ‘Safety of care’ or ‘Readmissions’. The revised CMS rating system ameliorates previous partial inconsistencies in association between Google ratings and component scores of HCSR. CONCLUSIONS Overall, crowd sourced Google hospital ratings are informative about expert CMS hospital quality ratings and several individual quality components that are easier for patients to evaluate. Therefore, hospitals should not expect improvements in quality metrics that require expertise to assess such as safety of care and readmission to result in improved Google star ratings. Hospitals can benefit from using crowd-sourced ratings as timely, easily available, and dynamic indicators of their quality performance.