Towards Gaze-contingent Visualization of Real-time 3D Reconstructed Remote Scenes in Mixed Reality.

Real-time 3D perception through RGB-D cameras and its rendering inside modern mixed reality (MR) devices have been shown to enhance a user’s sense of presence and immersion in a remote scene. In telepresence applications, it allows the operator to control a remote robot with full immersion. However, the streaming of such real-time 3D data from RGB-D cameras, especially real-time dense 3D reconstructed scenes, suffer from network latency and throughput limitations that can negatively impact user experience. The human visual system can serve as an inspiration to address this problem. Human eyes have their sharpest visual acuity at the center of their field-of-view, which falls off at the periphery. This research proposes a framework that utilizes this concept to intelligently sample, process, stream, and render of dense 3D reconstructed scenes in MR for immersive visualization. The proposed system, termed as Gaze-contingent Visualization System, shows significant reductions in latency and throughput needs, up to 80% in both. The innovative system improves the state-of-the-art in immersive visualization of remote real-time 3D reconstructed scenes.