Real-Time Semantic Segmentation of Spherical Images for Automotive Applications

Abstract Recent advancements in autonomous driving technology have resulted in a growing need for robust algorithms that can effectively detect, recognize, and segment objects in the surrounding environment. Semantic segmentation systems, which classify each pixel of an image into a class, have become a key component in achieving complete scene understanding, a crucial task for self-driving vehicles. This work focuses on utilizing deep learning methods to perform semantic segmentation on spherical images, a particular representation of fisheye images, characterized by a large field of view and strong distortion, which can result in performance degradation on convolutional neural networks (CNNs) trained on pinhole images. To address this issue, we propose the use of spherical kernels, which adapt their sampling locations according to the spherical projection, as an alternative to standard convolutional kernels, in real-time CNNs. We demonstrate that this approach outperforms the use of the regular convolution operation. To showcase the effectiveness of this method, we generate three spherical image datasets with varying characteristics.