Evaluating Visual and Auditory Substitution of Tactile Feedback During Mixed Reality Teleoperation

Mixed reality (MR) technology has shown enormous potential for real-time human-robot teleoperation. To provide the user with sensor feedback, typically extensive instrumentation is required, such as wearable devices. In this paper, we introduce an MR human-robot teleoperation system based on the Microsoft HoloLens 2 (HL2). The user can directly control the end-effector pose and the gripper of an arbitrary robot arm via hand tracking. Additionally, tactile information can be perceived without wearing data gloves. Therefore, the tactile information gathered by the robot during interaction with objects is substituted and presented to the user visually and auditory. We conducted a pilot user study to analyze the system’s usability and the effects of substituted tactile feedback during typical teleoperation tasks. The results show that the system is applicable for teleoperation as the users reached an 87.1% success rate in the performed manipulation task. The users were satisfied with the easy-to-use teleoperation interface and reported a preference for multimodal tactile substitution modes. Regarding the performance, an improvement in the average applied force could be observed when providing tactile feedback.