A Learning-based Framework for Multi-View Instance Segmentation in Panorama

The application of panoramas in computer vision has received a lot of attention due to its ability to represent information about the surrounding environment. Instance segmentation on panoramas can make machines better understand 3D scenes. However, there are few efforts have been made on detecting the instance for panoramas. The main challenge is that in a panorama, objects are subject to three variations: geometric distortion, edge discontinuity and minification of objects. To address the above issues, we propose an instance segmentation method for panoramas based upon the multi-view fusion. First, a set of sub-views are sampled from a panorama by a rectilinear projection, and then the Cascade Mask R-CNN is employed to perform instance segmentation on each sub-view. Subsequently, we reverse mapping the obtained instance segmentation results of sub-views back to the panorama. Finally, we combine the complementary advantages of large and small fields of view, and merge the instance segmentation results of the entire panorama with the reorganized instance segmentation results to obtain high-quality instance segmentation results for panoramas. The quantitative experiments illustrate that our proposed method obtains higher Mean Average Precision (0.28) than existing benchmark methods.