Children’s play and problem-solving in motion-based learning technologies using a multi-modal mixed methods approach

Motion-Based Learning Technologies (MBLT) offer a promising approach for integrating play and problem-solving behaviour within children’s learning. The proliferation of sensor technology has driven the field of learning technology towards the development of tools and methods that may benefit from the produced Multi-Modal Data (MMD). Such data can be used to uncover cognitive, affective and physiological processes during learning activities. Combining MMD with more traditionally exercised assessment tools, such as video content analysis, provides a more holistic understanding of children’s learning experiences and has the potential to enable the design of educational technologies capable of harmonising children’s cognitive, affective and physiological processes, while promoting appropriately balanced play and problem-solving efforts. However, the use of an MMD mixed methods approach that combines qualitative and MMD data to understand children’s behaviours during engagement with MBLT is rather unexplored. We present an in-situ study where 26 children, ages 10–12, solved a motion-based sorting task for learning geometry. We continuously and unobtrusively monitored children’s learning experiences using MMD collection via eye-trackers, wristbands, Kinect joint tracking, and a web camera. We devised SP3, a novel observational scheme that can be used to understand children’s solo interactions with MBLT, and applied it to identify and extract children’s evoked play and problem-solving behaviour. Collective analysis of the MMD and video codes provided explanations of children’s task performance through consideration of their holistic learning experience. Lastly, we applied predictive modelling to identify the synergies between various MMD measurements and children’s play and problem-solving behaviours. This research sheds light on the opportunities offered in the confluence of video coding (a traditional method in learning sciences) and MMD (an emerging method that leverages sensors proliferation) for investigating children’s behaviour with MBLT.