Evaluating Usability and User Experience of AR Applications in VR Simulation

Validating an augmented reality application in a virtual reality simulation can offer many advantages compared to testing in real conditions and can speed up development processes. With such a simulation, developers and designers do not need to have constant physical access to the real place. They can save physical navigation, experiment with different kinds of devices and isolate testing parameters. While the validity of functional testing in virtual reality simulations is not particularly challenged, the validity of such simulations to evaluate user experience and usability, similarly as in real conditions, still needs to be assessed. We then conducted a user study to explore the validity of evaluating these criteria with a virtual reality simulation tool and the importance of simulation fidelity for that purpose. Indeed, we also seek to determine whether it is necessary to simulate the limited field of view of augmented reality glasses and if the simulation can take place in a virtual world that is not a replica of the real targeted environment. To do so, we have developed an augmented reality application for smart-homes where a user can interact with different connected objects. One group of users performed the experiment in the real place with augmented reality glasses and three other groups performed the same experiment in virtual reality with various simulation conditions (field of view and environment). Users’ subjective feedback and quantitative results only highlight very few differences between real-world conditions and simulation in virtual reality, whatever the simulation parameters used. These results suggest the interest in using virtual reality simulation to evaluate an augmented reality application but should be confirmed on other use cases and interaction tasks.