Audio Visual Synchronization of Rhythm

The use of sound feedback to assist in learning and correcting motor skills has some innate advantages. For example, sound leaves a person's eyes free for important tasks like avoiding obstacles, and ample psychological evidence shows a connection between the synthesis and perception of motion and sound. When motion is periodic, as it is with most forms of human locomotion, the sound feedback should also be periodic, i.e., rhythmic and synchronized to a person's motion. We present a method to track an athlete performing a known movement in real-time using an RGB-D camera. We use a Kalman filter in conjunction with iterative closest point to synchronize incoming data against a known movement pattern and produce a musical rhythm. The athlete and the rhythm move in phase with one another. We call this process rhythmic sonification. We demonstrate our method by tracking and rhythmically sonifying an advanced in-line speed skating movement called double-push. Rhythmically sonified feedback provides a new opportunity to teach and reinforce this and other athletic skills.