iSense3D: A real-time viewpoint-aware 3D video synthesis system

In this paper, a real-time 3D video synthesis system is proposed. The system achieves more real 3D effect by dynamically adapts the synthesized view to user's viewpoint. There are two major parts: 6D viewpoint parameter extraction, and real-time Free-Viewpoint View Synthesis(FVVS). 6D viewpoint parameter extraction is done by 3D object tracking over image and depth. Various techniques for FVVS are proposed and implemented on GPU to achieve real-time. The system is demonstrated on multi-core system with programmable GPU. Real-time performance up to 1280×720p with 30fps is achieved.