A speech recognition-based interaction approach applying to immersive virtual maintenance simulation

Maintenance simulation plays an important role in maintainability design and analysis of products. The immersive virtual maintenance simulation, with motion capture devices driving the virtual personnel to perform the same action as the operator does, can make the maintenance simulation more real and valuable. However, because of a lack of interaction approach between the operator and simulation system, the operator cannot drive the virtual personnel to interact with 3D entities in virtual environment, and consequently, interactions like grasping and carrying become impossible. Therefore, three is a great need to find a way to deal with the plight. In this paper, a speech recognition-based interaction approach is put forward for immersive virtual maintenance simulation to help the operator to control the interaction process with voice command. Based on analysis on maintenance action, an abstract mechanism for instruction language is developed. Next, the paper gives the simulation framework for speech recognition-based immersive virtual maintenance simulation system. Through Microsoft Speech SDK and Secondary Development Technology of DELMIA, a functional simulation system is built and used to carry out an engine maintenance simulation. The result of this application shows the efficiency and effectiveness of the speech recognition-based interaction approach proposed for immersive virtual maintenance simulation.