MTP: Multi-hypothesis Tracking and Prediction for Reduced Error Propagation

There has been tremendous progress in the development of individual modules of the standard perception-prediction-planning robot autonomy stack. However, the principled integration of these modules has received less attention, particularly in terms of cascading errors. In this work, we both characterize and address the problem of cascading errors, focusing on the coupling between tracking and prediction. First, we comprehensively evaluate the impact of tracking errors on prediction performance with modern tracking and prediction methods on real-world data. We find that prediction methods experience a significant (even order of magnitude) drop in performance when consuming tracked trajectories as inputs (typical in practice), compared to the idealized setting where ground truth past trajectories are used as inputs. To address this issue, we propose a multi-hypothesis tracking and prediction framework. Rather than relying on a single set of tracking results for prediction, we simultaneously reason about multiple sets of tracking results, thereby increasing the likelihood of including accurate tracking results as inputs to prediction. We show that our framework(1) improves overall prediction performance over the standard single-hypothesis tracking-prediction pipeline by up to 34.2% on the nuScenes dataset, with even more significant improvements (up to similar to 70%) when restricting evaluation to challenging scenarios involving identity switches and fragments, all with an acceptable computation overhead.