Implementation of a Holistic Review Process of US Allopathic Medical Students Eliminates Non-Comparable Metrics and Bias in General Surgery Residency Interview Invitations

OBJECTIVE: Use of traditional scoring metrics for resi-dency recruitment creates racial and gender bias. In addi-tion, widespread use of pass/fail grading has led to noncomparable data. To adjust to these challenges, we developed a holistic review (HR) rubric for scoring resi-dency applicants for interview selection. DESIGN: Single-center observational study comparing the proportion of underrepresented in medicine (URM) students and their United States Medical Licensing Exam (USMLE) scores who were invited for interview before (2015-2020) and after (2022) implementation of a holis-tic review process. SETTING: General surgery residency program at a ter-tiary academic center. PARTICIPANTS: US allopathic medical students applying for general surgery residency. RESULTS: After initial screening, a total of 1514 allo-pathic applicants were narrowed down to 586 (38.7%) for HR. A total of 52% were female and 17% identified as URM. Based on HR score, 20% (118/586) of applicants were invited for an interview. The median HR score was 11 (range 4-19). There was a fourfold higher coefficient of variation of HR scores (22.3; 95% CI 21.0-23.7) com-pared to USMLE scores (5.1; 95% Cl 4.8-5.3), resulting in greater spread and distinction among applicants. There were no significant differences in HR scores between genders (p = 0.60) or URM vs non-URM (p = 0.08). There were no significant differences in Step 1 (p = 0.60) and 2CK (p = 0.30) scores between those who were invited to interview or not. On multivariable analysis, USMLE scores (OR 1.01; 95% CI 0.98-1.03), URM status (OR 1.71 95% CI 0.98-2.92), and gender (OR 0.94, 95% CI 0.60-1.45) did not predict interview selection (all p > 0.05). There was a meaningful increase in the percentage of URM interviewed after HR implementation (12.9% vs 23.1%, p = 0.016). CONCLUSION: The holistic review process is feasible and eliminates the use of noncomparable metrics for sur-gical applicant interview invitations and increases the percentage of URM applicants invited to interview. (J Surg Ed 80:1536-1543. (c) 2023 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) ABBREVIATIONS: URM, underrepresented in medicine USMLE, United States Medical Licensing Exam ACGME, the Accreditation Council for Graduate Medical Educa-tion APDS, Association of Program Directors in Surgery ABSITE, American Board of Surgery In-Service Training Exam IMG, International Medical Graduates DO Doctor of Osteopathic Medicine