Real-time Omnidirectional Depth Perception Based on Multi-view Wide-angle Vision System

Omnidirectional depth matters significantly to the field of environmental 3D perception. Currently, the learning methods for inferring depth from visual information primarily focus on accuracy improvement at the expense of computational speed and cost, which makes them unsuitable for some real-time applications like robotics. In this paper, we present a real-time and lightweight deep learning network to predict omnidirectional depth from four wide-angle camera images, which includes a multi-scale feature extractor and a two-stage cost aggregation architecture to effectively integrate feature information. Moreover, we propose a synthetic urban dataset for learning omnidirectional depth. It relies on four lenses, each with the field-of-view (FOV) of 180 degrees, and a panoramic lens to render multi-view images and ground-truth depth maps on the Blender platform. Plus, the GAN network is employed to enhance the dataset diversity by producing different weather samples. Finally, the whole network is trained and evaluated on our dataset, and experimental results demonstrate the effectiveness of the dataset and the feasibility of the proposed method. It can achieve a trade-off between model accuracy, computational speed and complexity, making it useful for real-time applications.