42: Milestone feedback and TIPreport: implementation and construct validity assessment

The study objectives were to describe implementation of TIPreport for milestone feedback and to initiate construct validity testing of TIPreport for milestone assessment. OBGYN and FPMRS learners and teachers from a convenience-based site selection of programs, used the web-based program TIPreport (TIP standing for “Training for Independent Practice”), for work-place feedback on ACGME-required milestone subcompetencies or “sets”. To characterize implementation, we described feedback frequency for each milestone set and for each clinical venue in which learner performance was observed. Performance of senior learners, PGY4s (residents), and PGY7s (fellows), was compared to that of junior learners to begin the process of construct validity testing for TIPreport. To minimize type I error in comparing senior to junior learner assessments, a threshold of 10 observations per comparison group was set. From November 1, 2014 – October 31, 2017, 12 OBGYN residency and 7 FPMRS fellowship programs participated. There were 480 unique learners (n=444 OBGYN, n=36 FPMRS) and 376 unique teachers (n=343 OBGYN, n=33 FPMRS). A total of 6138 milestone feedback encounters were recorded (n=5293 OBGYN and n=845 FPMRS). Feedback was recorded for each of the 28 OBGYN and 23 FPMRS milestone sets across all 6 ACGME competency areas. For OBGYN, feedback was most commonly reported for the milestone competency areas of Patient Care [n=2364 (45%)], Interpersonal and Communication Skills [n=1094 (21%)], and Medical Knowledge [n=1085 (20%)]. Competency areas most frequently reported on for FPMRS were Medical Knowledge [n=282 (33%)], Patient Care [n=258 (31%)], and Interpersonal and Communication Skills [n=117 (14%)]. Venues most often associated with OBGYN milestone feedback included the clinic [n=2084 (40%)], labor and delivery [n=1037 (20%)], and the operating room [n=844 (16%)]. FPMRS milestone feedback was reported most frequently in the clinic [n=560 (66%)] followed by operating room [n=141 (17%)]. For our primary outcome, comparing senior to junior learner milestone set assessments, sufficient numbers of observations were recorded for 25 of 28 (89%) of the OBGYN milestone sets. For 21 of 25 (84%) of these, PGY4 performance was rated significantly higher than that of PGY1s. For FPMRS, there were sufficient observations for 13 of 23 (57%) of milestone sets. In 6 of 13 (46%) of these, PGY7 performance was rated significantly higher than for PGY5s. TIPreport appeared to detect differences between senior and junior learners, for the majority of compared assessments for OBGYN, and for the near-majority for FPMRS. These data support the emerging construct validity of TIPreport for milestone assessment.