Live4D: A Real-time Capture System for Streamable Volumetric Video

Volumetric video holds promise for virtual and augmented reality (VR/AR) applications but faces challenges in interactive scenarios due to high hardware costs, complex processing and substantial data streams. In this paper, we introduce Live4D, a cost-effective, real-time volumetric video generation and streaming system using an RGB-only camera setup. We propose a novel deep implicit surface reconstruction algorithm, that combined neural signed distance field with observed truncated signed distance field to generate the watertight meshes with low latency. Moreover, we achieve a robust non-rigid tracking method that provides temporal stability to the meshes while resisting tracking failure cases. Experimental results show that Live4D achieves a performance of 24fps using mid-range graphic cards and exhibits an end-to-end latency of 95ms. The system enables live streaming of volumetric video within a 20Mbps bandwidth requirement, positioning Live4D as a promising solution for real-time 3D vision content creation in the growing VR/AR industry.