Should non-expert clinician examiners be used in objective structured assessment of communication skills among final year medical undergraduates?

Background: Adoption of the objective structured clinical examination may be hindered by shortages of clinicians within a specialty. Clinicians from other specialties should be considered as alternative, non-expert examiners. Aims: We assessed the inter-rater agreement between expert and non-expert clinician examiners in an integrated objective structured clinical examination for final year medical undergraduates. Methods: Pairs of expert and non-expert clinician examiners used a rating checklist to assess students in 8 oral communication stations, representing commonly encountered scenarios from medicine, paediatrics, and surgery. These included breaking bad news, managing an angry relative, taking consent for lumbar puncture; and advising a mother on asthma and febrile fits, and an adult on medication use, lifestyle changes and post-suture care of a wound. 439 students participated in the OSCE (206 in 2005, 233 in 2006). Results: There was good to very good agreement (intraclass coefficient: 0.57-0.79) between expert and non-expert clinician examiners, with 5 out of 8 stations having intraclass coefficients >= 0.70. Variation between paired examiners within stations contributed the lowest variance to student scores. Conclusion: These findings support the use of clinicians from other specialties, as 'non-expert' examiners, to assess communication skills, using a standardized checklist, thereby reducing the demand on clinicians' time.