ImFusion: Boosting Two-Stage 3D Object Detection via Image Candidates

Multi-modal fusion methods combine the advantages of both point clouds and RGB images to boost the performance of 3D object detection. Despite the significant progress, we find that existing two-stage multi-modal fusion methods suffer from the 3D proposal missing in the first stage and projected-style feature fusion mechanism. To solve these problems, we propose a two-stage multi-modal feature fusion network, which improves the recall rate of hard targets in the first stage of network with pseudo 3D proposals generated from image candidates. Then, considering the complementary information between similar image foreground features across multiple objects, we design a multi-modal cross-target fusion module to pay more attention to the foreground objects. It enables a 3D proposal can aggregate the semantic features of multiple image candidates belonging to the same category. Finally, these enhanced fused proposals are processed in the second stage to further boost the performance of 3D detector. Experimental results on SUN RGB-D and KITTI datasets show the effectiveness of our proposed method.