Integrated Debugging of Large Modular Robot Ensembles

Creatively misquoting Thomas Hobbes, the pro- cess of software debugging is nasty, brutish, and all too long. This holds all the more true in robotics, which frequently involves concurrency, extensive nondeterminisism, event-driven components, complex state machines, and difficult platform limitations. Inspired by the challenges we have encountered while attempting to debug software on simulated ensembles of tens of thousands of modular robots, we have developed a new debugging tool particularly suited to the characteristics of highly parallel, event- and state-driven robotics software. Our state capture and introspection system also provides data that may be used in higher-level debugging tools as well. We report on the design of this promising debugging system, and on our experiences with it so far. I. I NTRODUCTION As ever-more-powerful embedded computers and PCs have become ubiquitous, robotics software designers have exploited this additional computational horsepower to per- form increasingly complex tasks, including vision process- ing, online learning, and sophisticated planning operations. Many modern robot designs rely on multiple software mod- ules executing as concurrent threads, either time-sliced or spread across multiple cooperating CPUs. While this in- creased software complexity can help enable more sophisti- cated functionality, it comes at the cost of a greater likelihood of implementation or algorithmic errors as well as bugs due to concurrency, nondeterminism, and subtle interactions between the multiple threads of execution. Debugging such multithreaded robotics software is challenging even when it is run on a simulated platform, and yet more difficult on the actual embedded hardware itself. Software development for modular and metamorphic robotic systems compounds these issues, as it involves sys- tems of many robots, each with one or more threads of execution. Programs for even modest modular robot ensem- bles may involve hundreds of concurrently executing threads. The Claytronics project (9) envisions using thousands to mil- lions of tiny modular robots (catoms, short for "Claytronics atoms") that dynamically reconfigure to render and animate 3D scenes and structures. Our current simulations routinely involve over 50,000 robot modules (each with at least one independent stream of execution), and in the near future we expect to simulate over a half-million modules. Debugging software and algorithms at this scale of concurrency is a daunting task.