Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning

We apply multi-agent deep reinforcement learning (RL) to train end-to-endrobot soccer policies with fully onboard computation and sensing via egocentricRGB vision. This setting reflects many challenges of real-world robotics,including active perception, agile full-body control, and long-horizon planningin a dynamic, partially-observable, multi-agent domain. We rely on large-scale,simulation-based data generation to obtain complex behaviors from egocentricvision which can be successfully transferred to physical robots using low-costsensors. To achieve adequate visual realism, our simulation combines rigid-bodyphysics with learned, realistic rendering via multiple Neural Radiance Fields(NeRFs). We combine teacher-based multi-agent RL and cross-experiment datareuse to enable the discovery of sophisticated soccer strategies. We analyzeactive-perception behaviors including object tracking and ball seeking thatemerge when simply optimizing perception-agnostic soccer play. The agentsdisplay equivalent levels of performance and agility as policies with access toprivileged, ground-truth state. To our knowledge, this paper constitutes afirst demonstration of end-to-end training for multi-agent robot soccer,mapping raw pixel observations to joint-level actions, that can be deployed inthe real world. Videos of the game-play and analyses can be seen on our websitehttps://sites.google.com/view/vision-soccer .