Children's Play and Problem Solving in Motion-Based Educational Games: Synergies between Human Annotations and Multi-Modal Data

Identifying and supporting children's play and problem solving behaviour is important for designing educational technologies. This can inform feedback mechanisms to scaffold learning (provide hints or progress information), and assist facilitators (teachers, parents) in supporting children. Traditionally, researchers manually code video to dissect children's nuanced play and problem solving behaviour. Advancements in sensing technologies and their respective Multi-Modal Data (MMD), afford observation of invisible states (cognitive, affective, physiological), and provide opportunities to inspect internal processes experienced during learning and play. However, limited research combines traditional video annotations and MMD to understand children's behaviour as they interact with educational technology. To address this concern, we collected data from webcam, wristband, eye-trackers, and Kinect, as 26 children, aged 10-12, played a Motion-Based Educational Games (MBEG). Results showed significant differences in children's experience during play and problem solving episodes, and motivate design considerations aimed to facilitate children's interactions with MBEG.