Quality Improvement Of Multiple Choice Examinations. In Psychiatry, Psychosomatic Medicine, Psychotherapy, And Neurology

We describe a continuous improvement process in planning, performance, and evaluation of multiple choice examination questions in psychiatry, neurology, psychosomatic medicine, and psychotherapy. We analyzed 640 multiple choice questions of 1,419 students during a period of 4 years. Crucial changes concerned the abolishment of problematic question types, implementation of validated new question formats, extension of case-based questions, elongation of question stems, quantitive evaluation of item difficulty, discriminatory value, and the introduction of a peer review system. Consequences of these improvements were greater item difficulty (average 18%) and discriminatory value (average 67%) and reduced post hoc analysis times. Introduction of peer reviews resulted in longer preparation time, which was however appreciated by the peers due to a clear improvement in item quality.