Learned Dynamic Guidance for Depth Image Reconstruction
IEEE Transactions on Pattern Analysis and Machine Intelligence(2020)
摘要
The depth images acquired by consumer depth sensors (e.g., Kinect and ToF) usually are of low resolution and insufficient quality. One natural solution is to incorporate a high resolution RGB camera and exploit the statistical correlation of its data and depth. In recent years, both optimization-based and learning-based approaches have been proposed to deal with the guided depth reconstruction problems. In this paper, we introduce a weighted analysis sparse representation (WASR) model for guided depth image enhancement, which can be considered a generalized formulation of a wide range of previous optimization-based models. We unfold the optimization by the WASR model and conduct guided depth reconstruction with dynamically changed stage-wise operations. Such a guidance strategy enables us to dynamically adjust the stage-wise operations that update the depth image, thus improving the reconstruction quality and speed. To learn the stage-wise operations in a task-driven manner, we propose two parameterizations and their corresponding methods: dynamic guidance with Gaussian RBF nonlinearity parameterization (DG-RBF) and dynamic guidance with CNN nonlinearity parameterization (DG-CNN). The network structures of the proposed DG-RBF and DG-CNN methods are designed with the the objective function of our WASR model in mind and the optimal network parameters are learned from paired training data. Such optimization-inspired network architectures enable our models to leverage the previous expertise as well as take benefit from training data. The effectiveness is validated for guided depth image super-resolution and for realistic depth image reconstruction tasks using standard benchmarks. Our DG-RBF and DG-CNN methods achieve the best quantitative results (RMSE) and better visual quality than the state-of-the-art approaches at the time of writing. The code is available at https://github.com/ShuhangGu/GuidedDepthSR.
更多查看译文
关键词
weighted analysis sparse representation model,guided depth image enhancement,previous optimization-based models,WASR model,conduct guided depth reconstruction,dynamically changed stage-wise operations,guidance strategy,reconstruction quality,Gaussian RBF nonlinearity parameterization,CNN nonlinearity parameterization,DG-RBF,DG-CNN methods,optimal network parameters,paired training data,optimization-inspired network architectures,guided depth image super-resolution,realistic depth image reconstruction tasks,learned dynamic guidance,consumer depth sensors,insufficient quality,high resolution RGB camera,guided depth reconstruction problems
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要