Proposal : Audio-Material Object Reconstruction for Multimodal Interaction

Virtual environments are utilized in user interaction scenarios to achieve diverse goals. Interactive training simulations enable low-risk practice of high-risk tasks, such as performing surgery or piloting an airplane. Immersive story-driven video games let users interact with another environment or become involved in an engaging narrative. Emerging social applications can unite multiple users from around the world in one virtual location such that the users actually feel as through they are in the same space. In these scenarios, the user should be able to forget their presence in the real world and temporarily experience a sense of presence in the virtual environment. If the user remembers that the virtual environment is fake, they experience a break in presence. When this happens, the emotional weight of the virtual environment may be reduced, making it less effective at its intended goal. By better maintaining a user’s sense of presence, we can improve the quality of their experience. Training simulations will feel more lifelike, video games will convey more powerful emotions, and social interactions with other users will flow more naturally. An undesirable break in presence can be triggered by many situations, but one common issue is when a user’s expectations about their senses are violated. Currently, virtual environments most commonly recreate the senses of sight and hearing. The visuals and audio of the real world are relatively easily replaceable with those of a virtual world. A virtual reality (VR) headset such as the Oculus Rift or the HTC Vive replaces the sense of sight, while high-quality headphones (sometimes built into the VR headset) replace the sense of hearing. A few examples of VR-enabled environments are shown in Figure 1. Humans also rely heavily on the sense of touch, but virtual environments are limited by current hardware, which cannot effectively replace our complete sense of touch. A break in presence may occur if a single sense is inconsistent with a