An empirical evaluation of the User Engagement Scale (UES) in online news environments

We examined the robustness of the User Engagement Scale (UES).Three studies were conducted in Canada and the United Kingdom.The UES sub-scales were reliable across three samples of online news browser.A four-factor structure, rather than six, may be more appropriate.The UES differentiated between online news sources and experimental conditions. Questionnaires are commonly used to measure attitudes toward systems and perceptions of search experiences. Whilst the face validity of such measures has been established through repeated use in information retrieval research, their reliability and wider validity are not typically examined; this threatens internal validity. The evaluation of self-report questionnaires is important not only for the internal validity of studies and, by extension, increased confidence in the results, but also for examining constructs of interest over time and across different domains and systems.In this paper, we look at a specific questionnaire, the User Engagement Scale (UES), for its robustness as a measure. We describe three empirical studies conducted in the online news domain and investigate the reliability and validity of the UES. Our results demonstrate good reliability of the UES sub-scales; however, we argue that a four-factor structure may be more appropriate than the original six-factor structure proposed in earlier work. In addition, we found evidence to suggest that the UES can differentiate between systems (in this case, online news sources) and experimental conditions (i.e., the type of media used to present online content).