A Direct Observation Form for Evaluation of the Psychiatric Interview: Pilot Testing During the Psychiatry Clerkship

Objectives The authors evaluated the validity of a structured direct observation form (DOF) to assess medical student performance on psychiatric interviews for use in the psychiatry clerkship. Methods One hundred and forty-eight third-year medical students were evaluated by two DOFs completed by a supervising resident, fellow, attending, or another team member. One DOF was completed early (time 1) and the other late (time 2) during a 4-week psychiatry clerkship. Results The DOF showed good internal consistency (Cronbach’s alpha = 0.88–0.89). DOFs submitted at time 2 were positively associated with end-of-course clinical grades ( p < 0.001); this association remained significant while controlling for time of academic year the course was completed, rater rank, complexity of the case, and difficulty of the interview (time 2 p < 0.001). Mean scores from the DOF were associated with the time of year students took the course with students assessed early in the academic year having lower average scores ( p -values = 0.01 at time 1, 0.002 at time 2). Scores on time 1 DOFs were positively associated with rater rank ( p = 0.005; residents gave higher scores than faculty). DOFs also correlated with an Entrustable Professional Activity (EPA) assessment of the interview (time 1 r = 0.76, p < 0.001; time 2 r = 0.79, p < 0.001), but not with shelf exam scores (time 1 r = 0.10, p = 0.24; time 2 r = 0.11, p = 0.21). Conclusions A brief structured form evaluating medical student performance on psychiatric interviews provided valid information about performance by third-year medical students during the psychiatry clerkship.