Implications of Excluding the Assessment of Thesis from Subspecialty Certifying Examinations on Pass Rates and Reliability.

The American Board of Obstetrics and Gynecology (ABOG) certification for the 4 obstetrics and gynecology subspecialties (female pelvic medicine and reconstructive surgery [FPMRS], gynecologic oncology [GO], maternal-fetal medicine [MFM], and reproductive endocrinology and infertility [REI]) is a 2-step process that requires passing a computer-based qualifying examination and an oral certifying examination. Each fellow completes at least 12 months of research experiences during fellowship training and completes a thesis in the subspecialty area. Before 2022, the subspecialty certifying examinations included an assessment of each candidate’s thesis and an assessment of knowledge, skills, and judgment in the subspecialty by using standardized structured cases and a candidate’s practice case lists (Table).TableStructure change of the subspecialty certifying examination from 2021 to 2022DurationSection 1Section 2Section 330 min5 structured cases5 structured cases2021: Thesis—weighted 5 times2022: Thesis replaced with 5 structured cases (combination of research and evidence-based medicine and clinical standardized cases)30 minCase list (weighted)Case list (weighted)Case list (weighted)Chadha. Impact of excluding thesis from certifying examinations. Am J Obstet Gynecol 2023. Open table in a new tab Chadha. Impact of excluding thesis from certifying examinations. Am J Obstet Gynecol 2023. During the COVID-19 pandemic, fellowship training programs across the United States faced various disruptions in clinical, didactic, and research activities. Obstetrics and gynecology subspecialty programs were similarly affected.1Butler B.M. Biller D.H. Effect of COVID-19 on female pelvic medicine and reconstructive surgery fellowship education and training.Female Pelvic Med Reconstr Surg. 2022; 28: 336-340Crossref PubMed Scopus (2) Google Scholar,2Ghidei L. Gannon A. Schutt A. The impact of COVID-19 on REI fellowship educational experience.J Assist Reprod Genet. 2021; 38: 1163-1169Crossref PubMed Scopus (2) Google Scholar Because each subspecialty’s program requirements and certification standards include a minimum of 12 months of research and completion of a related thesis, the assessment of the thesis in the certifying examination could disadvantage fellows who experienced disruption in research and thesis development. Thus, to aid process fairness, the assessment of the thesis was excluded from the 4 certifying examinations in the year 2022.3American Educational Research AssociationAmerican Psychological Association, National Council on Measurement in Education.Standards for educational and psychological testing. American Educational Research Association, Washington, DC2014Google Scholar The assessment of the thesis had been included in the 2021 subspecialty examinations. This study aimed to assess the implications of the exclusion of the assessment of the thesis on the pass rates and reliability of scores. The 2022 ABOG subspecialty certifying examinations were administered virtually between April 4, 2022, and April 7, 2022, and did not include an assessment of the thesis. In the 2022 examinations, the thesis section was replaced with structured cases that assessed research design, methodology, evidence-based medicine, and clinical topics, as determined appropriate by our job task analysis. The scoring of this section’s cases contributed the same weight in 2022 as the actual assessment of the thesis did in 2021. The difference in the subspecialty certifying examination structure from 2021 and 2022 is displayed in the Table. Each candidate’s thesis was still reviewed by the respective ABOG subspecialty division members using a standardized rubric to ensure that it met ABOG requirements to be eligible for the certifying examination. The pass rates of first-time and overall (first-time and repeat) takers in the 2022 certifying examinations were compared with that of those in the 2021 certifying examinations using the 2-proportion z test in R.4R Core TeamR: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria2019Google Scholar P values of <.05 were considered significant. Furthermore, the reliability of scores for the 2022 certifying examination was compared with that of the 2021 certifying examination as the scoring methodology was the same for these 2 certifying examinations.5Linacre J.M. Many-facet Rasch measurement.2nd ed. MESA Press, San Diego, CA1994Google Scholar The observed pass rates are displayed in the Figure. There was no significant difference in the pass rates of overall and first-time takers between the 2022 and 2021 certifying examinations. The reliability of scores was the same for REI and MFM across 2 years (0.97 for both). However, from 2021 to 2022, the reliability of scores increased slightly from 0.96 to 0.97 in GO and 0.94 to 0.96 in FPMRS. Replacing the assessment of the thesis with the assessment of knowledge, skill, and judgment using standardized structured cases did not considerably alter the pass rates or the reliability of scores of overall and first-time takers compared with that of the preceding takers of the subspecialty certifying examinations. Standardized structured cases seemed to be a reasonable alternative to the assessment of the thesis in the certifying examinations. During continued surveillance of the disruption of the COVID-19 pandemic to fellowship training programs, these findings may aid future decision-making regarding the structure of subspecialty certifying examinations.