Challenges and Workarounds of Conducting Augmented Reality Usability Tests Remotely a Case Study.

It has been demonstrated that virtual prototypes can be used productively in usability testing. Virtual prototypes, Augmented Reality (AR) in particular, has received some focus as it combines both the real and virtual worlds and because platforms capable of supporting AR have become more ubiquitous. This paper aims to explore the challenges of using AR to conduct usability testing in a remote delivery format. It presents a use case examination of the use of an AR tool to gather usability assessment data of a design prototype in a remote (distanced) format. An interview was conducted with those involved in conducting the test to understand issues related to the utilization of hardware provided by the usability test taker; the use of screen sharing to communicate instructions; challenges to efficiently gathering accurate observational data; and issues related to variability in hardware setup, consistency of the AR experience, and other environmental factors. The results suggest that an approach for utilizing AR for remote usability testing was feasible, but many challenges and frustrations exist. Some of these related to communication with the participants, lack of motivation, difficulties extrapolating the prototype to a real setting, and technical issues. Usability test participants were often more interested in simply completing assigned tasks rather than exploring the tool or product in more detail that often happens in person. Other observations included some limitations in AR technology itself such as level of immersion (ability to accurately visualize the product). Challenges due to internet connectivity between the facilitator and tester at time affected the ability to effectively direct the testing session. Future work should investigate new tools and approaches for mitigating current challenges and experimentally compare in person versus remote testing outcomes.