GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time
arxiv(2024)
摘要
This paper presents GGRt, a novel approach to generalizable novel view
synthesis that alleviates the need for real camera poses, complexity in
processing high-resolution images, and lengthy optimization processes, thus
facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in
real-world scenarios. Specifically, we design a novel joint learning framework
that consists of an Iterative Pose Optimization Network (IPO-Net) and a
Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism,
the proposed framework can inherently estimate robust relative pose information
from the image observations and thus primarily alleviate the requirement of
real camera poses. Moreover, we implement a deferred back-propagation mechanism
that enables high-resolution training and inference, overcoming the resolution
constraints of previous methods. To enhance the speed and efficiency, we
further introduce a progressive Gaussian cache module that dynamically adjusts
during training and inference. As the first pose-free generalizable 3D-GS
framework, GGRt achieves inference at ≥ 5 FPS and real-time rendering at
≥ 100 FPS. Through extensive experimentation, we demonstrate that our
method outperforms existing NeRF-based pose-free techniques in terms of
inference speed and effectiveness. It can also approach the real pose-based
3D-GS methods. Our contributions provide a significant leap forward for the
integration of computer vision and computer graphics into practical
applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open
datasets and enabling real-time rendering for immersive experiences.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要