A multi-institutional study of surgical resident entrustability across multiple phases of care

Purpose There are concerns about the measurement burden that may be associated with Entrustable Professional Activities (EPAs) as well as the potential susceptibility of these ratings to influence from factors such as rater bias. These factors may threaten the ability of EPAs to discriminate resident entrustability between clinical phases of care. The purpose of this study was to understand the variation in entrustability ratings (observation only, direct supervision, indirect supervision, unsupervised practice, and supervising others) across pre-, intra-, and post-operative phases of clinical care. Any considerable overlap between phases may represent an opportunity for reducing measurement burden and for improving feasibility of EPAs. Methods As part of the American Board of Surgery EPA pilot study, faculty were asked to rate the entrustability of surgical trainees after direct observation of pre-, intra-, and/or post-operative phases of care. Ratings collected using a customized version of a smartphone-based workplace-based assessment tool (the SIMPL app) were included in this study. Descriptive statistics and linear mixed effects models were used to investigate the relationship among pre-, intra-, and post-operative EPA ratings for three operative EPAs: gallbladder disease, right lower quadrant pain, and inguinal hernia. Results Three hundred sixty five residents (210 faculty) in twelve programs were included in analyses. 10.6% of patient encounters included ratings for all three phases. In those encounters, ratings across all three phases of care agreed 54.0% of the time. When adjusted for PGY, trainee, faculty, and patient encounter, pre- and post-operative ratings were largely similar to each other, differing only at the PGY 3 level ( p = 0.01). Preoperative ratings differed from intraoperative ratings in the PGY 2, PGY 4, and PGY 5 cohorts (all p ≤ 0.01), and postoperative ratings differed from intraoperative ratings at all PGY levels (all p < 0.03). Faculty rater accounted for a large proportion of the variance (37%) in EPA rating across phases. Conclusions There is considerable overlap in entrustability ratings from the pre- and post-operative phases of care. Measurement burden may be reduced by more selectively assessing entrustability in these perioperative phases. Efforts to minimize the amount of variance in rating attributable to faculty rater are needed.