Towards Volumetric Video Realism in Extended Reality: Challenges and Opportunities

Advances in volumetric capture, compression, and rendering techniques have enabled the possibilities of telepresence in an extended reality (XR) environment. Live or pre-recorded volumetric video of an avatar can be streamed over the network and rendered at a client's XR environment, creating an illusion of spatial co-presence. In this keynote talk, I will first present a case for the importance of visual realism of volumetric video in such scenarios. I will then present existing approaches toward higher visual realism in volumetric video, dividing them into two categories: (i) approaches to achieve smoother motion through temporal up-sampling and (ii) approaches to obtain better details through spatial up-sampling. The former aims to achieve a rendering frame rate that is close to what human brains perceive; while the latter allows users to move closer to an avatar without losing its realism. The talk will also outline the trade-offs and limitations of the current up-sampling approaches. I will conclude the talk with my personal view on the research challenges and opportunities that the research community should confront to achieve a true-to-life XR experience.