Construct Validity of a Simple Laparoscopic Ovarian Cystectomy Model Using a Validated Objective Structured Assessment of Technical Skills

STUDY OBJECTIVE:To determine the construct validity and interrater reliability of a laparoscopic ovarian cystectomy simulator using a global rating scale. DESIGN:Prospective blinded observational study (Canadian Task Force classification II.3). SETTING:Academic teaching hospital. PARTICIPANTS:A total of 26 postgraduate year (PGY) 1 to 4 gynecology and obstetrics residents were recruited (15 junior residents, postgraduate year PGY 1-2 and 11 senior residents, PGY 3-4). INTERVENTION:We developed a simple, low-cost laparoscopic ovarian cystectomy simulator and incorporated it into our simulation curriculum. The simulation was directed at junior residents with instruction and immediate feedback in a scheduled simulation session once during the academic year. At the end of the year resident skills assessment, all levels were recorded with video using this model to assess the construct validity between junior and senior residents. Resident performance was later evaluated and scored by 2 blinded, experienced laparoscopists using a validated Objective Structured Assessment of Technical Skills (OSATS). MEASUREMENT AND MAIN RESULTS:Each resident received a unique identification number and the simulated laparoscopic ovarian cystectomy procedure was filmed during the end of the year assessment. Two blinded raters evaluated the video of each resident with the modified Global Rating Scale (OSATS), using 5 of the 7 domains (respect for tissue, time and motion, instrument handling, flow of operation and knowledge of specific procedure). An average of the 2 ratings was computed for each domain, and comparisons were made using the Mann-Whitney U test. Interrater reliability was calculated using the Kendall tau β correlation coefficient. Construct validity was determined by comparing the rank scores of the junior to senior residents in each domain. Construct validity and interrater reliability was demonstrated in all of the measured domains except for respect for tissue. CONCLUSION:This simple, low-cost model can be used to teach important laparoscopic ovarian cystectomy skills. Construct validity discriminating between junior and senior residents was demonstrated using this model.