The Agony of Choice: Acceptance, Efficiency, and Psychometric Properties of Questionnaires With Different Numbers of Response Options

Questionnaires are one of the most important tools in psychological assessment, yet the impact of different numbers of response options on psychometric properties of questionnaires is limited. This study extends existing research by analyzing respondents' acceptance of and the efficiency of different numbers of response options and replicate findings on reliability and validity. We studied these questions in 540 respondents who filled out the Big Five Inventory-2 and the Emotion Regulation Questionnaire. Two response options, 11 response options and the visual analog scale showed disadvantages in acceptance compared with the original number of response options. The completion time increased by 1.7s per item when moving from 2 to 11 response options. Cronbach's alpha (but not ordinal alpha based on polychoric covariance) was lowest for two response options. Validity was unaffected. Overall, compared with the typical choice of five or seven response options, fewer or more response options resulted in disadvantages.