Realistic training in VR using physical manipulation

We propose an interaction method for mixed reality with a direct application on training applications. It is based on the combination of two key ideas: providing real user embodiment within virtual spaces by segmenting user real hands and allowing the manipulation of physical objects, which may preserve or change their real appearance, in the immersive scenario. We deployed a functional prototype of this concept using Unity for the base scenario, a color-based algorithm for object segmentation and Aruco library for object tracking. Then, to take advantage of gamification, we built a five-minute escape room game designed to validate all components: real embodiment, real objects, augmented real objects and control devices. We carried out a thorough evaluation of our system with 53 users and a fair comparison scheme: each user had to play the escape room twice, once with real objects and hands, and once with purely virtual objects and avatars. After each run, users had to answer a survey derived from standard questionnaires measuring presence, embodiment and quality of experience. Some quantitative data related to performance were also extracted from the game runs. Results validate our hypothesis that this approach significantly improves key factors such as presence or embodiment with respect to the counterpart virtual solution. We foresee our solution to provide a significant advance in the implementation of virtual training applications.