CNN-MonoFusion: Online Monocular Dense Reconstruction Using Learned Depth from Single View

Online dense reconstruction is a major task of Augmented Reality (AR) applications, especially for realistic interactions like collisions and occlusions. Monocular cameras are most widely used on AR equipment, however existing monocular dense reconstruction methods have poor performance in practical applications due to the lack of true depth. This paper presents an online monocular dense reconstruction framework using learned depth, which overcomes the inherent difficulties of reconstruction for low-texture regions or pure rotational motions. Firstly, we design a depth prediction network combined with an adaptive loss, so that our network can be extended to train on mixed datasets with various intrinsic parameters. Then we loosely combine depth prediction with monocular SLAM and frame-wise point cloud fusion to build a dense 3D model of the scene. Experiments validate that our depth prediction from a single view reaches a state-of-the-art accuracy on different benchmarks, and the proposed framework can reconstruct smooth, surface-clear and dense models on various scenes with dedicated point cloud fusion scheme. Furthermore, collision and occlusion detection are tested using our dense model in an AR application, which demonstrates that the proposed framework is particularly suitable for AR scenarios. Our code will be publicly available along with our indoor RGB-D dataset at: https://github.com/NetEaseAI-CVLab/CNN-MonoFusion.