Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging
arxiv(2024)
摘要
Neural Radiance Fields (NeRFs) quickly evolved as the new de-facto standard
for the task of novel view synthesis when trained on a set of RGB images. In
this paper, we conduct a comprehensive evaluation of neural scene
representations, such as NeRFs, in the context of multi-modal learning.
Specifically, we present four different strategies of how to incorporate a
second modality, other than RGB, into NeRFs: (1) training from scratch
independently on both modalities; (2) pre-training on RGB and fine-tuning on
the second modality; (3) adding a second branch; and (4) adding a separate
component to predict (color) values of the additional modality. We chose
thermal imaging as second modality since it strongly differs from RGB in terms
of radiosity, making it challenging to integrate into neural scene
representations. For the evaluation of the proposed strategies, we captured a
new publicly available multi-view dataset, ThermalMix, consisting of six common
objects and about 360 RGB and thermal images in total. We employ cross-modality
calibration prior to data capturing, leading to high-quality alignments between
RGB and thermal images. Our findings reveal that adding a second branch to NeRF
performs best for novel view synthesis on thermal images while also yielding
compelling results on RGB. Finally, we also show that our analysis generalizes
to other modalities, including near-infrared images and depth maps. Project
page: https://mert-o.github.io/ThermalNeRF/.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要