Evaluating Surgical Complexity of Endoscopic Hysterectomy: An Inter-rater and Intra-rater Agreement Study of Novel Scoring Tool

Study Objective: Evaluate inter-rater and intrarater reliability of a novel scoring tool for surgical complexity assessment of endoscopic hysterectomy. Design: Validation study. Setting: Academic medical center. Participants: Total of 11 academic obstetrician-gynecologists with varying years of postresidency training, clinical practice, and surgical volumes. Interventions: Application of a novel scoring tool to evaluate surgical complexity of 150 sets of images taken in a standardized fashion before surgical intervention (global pelvis, anterior cul-de-sac, posterior cul-de-sac, right adnexa, left adnexa). Using only these images, raters were asked to assess uterine size, number, and location of myomas, adnexal and uterine mobility, need for ureterolysis, and presence of endometriosis or adhesions in relevant locations. Surgical complexity was staged on a scale of 1 to 4 (low to high complexity). Measurements and Main Results: Number of postresidency years in practice for participating surgeons ranged from 2 to 15, with an average of 8 years. A total of 8 obstetrician-gynecologists (72.7%) had completed a fellowship in minimally invasive gynecologic surgery. Six (54.6%) reported an annual volume of >50 hysterectomies. Raters reported that 95.4% of the images were satisfactory for assessment. Of the 150 sets of images, most were found to be stage 1 to 2 complexity (stage 1: 23.8%, stage 2: 41.6%, stage 3: 32.8%, stage 4: 1.8%). The level of inter-rater agreement regarding stage 1 to 2 vs 3 to 4 complexity was moderate (kappa = 0.49; 95% confidence interval [CI], 0.42-0.56). Moderate inter-rater agreement was also found between surgeon raters with an annual hysterectomy volume >50 (kappa = 0.49; 95% CI, 0.40-0.57) as well as between surgeon raters with fellowship experience (kappa = 0.50; 95% CI, 0.42-0.58). Intrarater agreement averaged 80.2% among all raters and also achieved moderate agreement (mean weighted kappa = 0.53; range, 0.38-0.72). Conclusion: This novel scoring tool uses clinical assessment of preintervention anatomic images to stratify the surgical complexity of endoscopic hysterectomy. It has rich and comprehensive evaluation capabilities and achieved moderate inter-rater and intrarater agreement. The tool can be used in conjunction with or instead of traditional markers of surgical complexity such as uterine weight, estimated blood loss, and operative time. (C) 2022 AAGL. All rights reserved.