The DARPA PerceptOR evaluation experiments

This article describes the conduct of six evaluation experiments for the Perception for Off-Road Robotics program. Key distinctions of the testing methodology include conduct of the experiments by a group independent from the developers, unrehearsed experiments that provide little advance knowledge of the test courses, and blind experiments that do not allow the system operators to see the test courses until testing has completed. The article presents quantified, objective performance metrics for the systems evaluated. The basis for evaluation is 296 runs traveling 130 km in 110 hr. The results show significant progress over the course of the program, reducing the Emergency-Stops per kilometer by a factor of 22, reducing the uplink data volume per unit distance by a factor of 46 and the downlink data volume per unit distance by a factor of 3. At the end of Phase III, typical performance in desert terrain by the most reliable system achieved travel speed of 66 cm/s covering 90% of the distance in autonomous mode.