Development, content validation, and reliability of the Assessment of Real-World Observational Studies (ArRoWS) critical appraisal tool

Purpose: The objective was to develop and test a pragmatic critical appraisal tool, the Assessment of Real-World Observational Studies (ArRoWS), to quickly and easily assess the quality of real-world evidence studies using electronic health records.Methods: The initial ArRoWS tool was developed by identifying items frequently found in existing validated assessment instruments and adapting these items to specifically assess real-world evidence studies. The tool was revised based on recommendations from an expert panel of 14 senior academic individuals specializing in epidemiology and content validity was measured. During March 2018-January 2019, 47 large, observational studies related to cardiometabolic medicine were identified through a search algorithm and assessed by three pairs of raters using the ArRoWS tool.Results: The final version of the ArRoWS had 16 items including nine core items and seven study design specific items with item-specific content validity indexes ranging from 0.64 to 1.00. The scale-level content validity index of the ArRoWS appraisal tool was 0.91. When the ArRoWS tool was pilot tested, the observed agreement between assessor pairs on whether the study provided high-quality real-world evidence was 85.7%, 68.8%, and 58.8%. The prevalence adjusted bias-adjusted kappa for the assessor pairs was 0.71, 0.38, and 0.18.Conclusion: The ArRoWS is a simple tool to standardize the assessment of real-world evidence studies. (c) 2020 Elsevier Inc. All rights reserved.