Interviewer judgment-making as a social practice: Insights from an institutional observational study in postgraduate selection

Abstract Background Interviewers’ judgements play a critical role in competency-based assessments for selection such as the multiple-mini-interview. ((MMI) Much of the published research focuses on the psychometrics of selection and the impact of rater subjectivity. Within the context of selecting for entry into specialty postgraduate training, we used an interpretivist and socio-constructivist approach to explore how and why interviewers make judgments in high stakes selection settings whilst participating in a multiple-mini-interview (MMI). Methods We explored MMI interviewers’ work processes through an institutional observational approach, based on the notion that interviewers’ judgements are socially constructed and mediated by multiple factors. We gathered data through document analysis, and observations of interviewer training, candidate interactions with interviewers, and interviewer meetings. Interviews included informal encounters in a large selection centre. Data analysis balanced description and explicit interpretation of the meanings and functions of the interviewers’ actions and behaviours. Results Three themes were developed from the data showing how interviewers make professional judgements, specifically by; ‘Balancing interplay of rules and agency,’ ‘Participating in moderation and shared meaning making; and ‘A culture of reflexivity and professional growth.’ Interviewers balanced the following of institutional rules with making judgment choices based on personal expertise and knowledge. They engaged in dialogue, moderation, and shared meaning with fellow interviewers which enabled their consideration of multiple perspectives of the candidate’s performance. Interviewers engaged in self-evaluation and reflection throughout, with professional learning and growth as primary care physicians and supervisors being an emergent outcome. Conclusion This institutional observational study of interviewers at work in an MMI gives rich insight into the social and contextual nature of judgment-based decision-making in the postgraduate selection process, and emergent outcomes relating to the professional growth of interviewers. Reframing interviewer judgement-making as a social practice provides ways to re-design approaches to interviewer training and selection tool design.