Challenging Autonomy with Combinatorial Testing

This paper describes an input space modelling and test generation method called CACTus (Challenging Autonomy with Combinatorial Testing) that creates a suite of ‘challenge scenarios’ for autonomous systems. Though the parameter space for autonomous systems is vast, CACTus helps to reduce the parameter space using combinatorial testing and by incorporating expert judgement into the formulation of the scenarios. The resulting scenarios can be executed on appropriate test infrastructure, such as simulators or hardware-in-loop testing. CACTus may be used by practitioners to exercise systems as part of efforts to obtain compliance with standards like ISO 21448 or UL 4600. The method is applied to generate test scenarios for the perception system of a commercial autonomous vehicle.