Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems.

While using online learning software, students demonstrate many reactions, various levels of engagement, and emotions (e.g. confusion, boredom, excitement). Having such information automatically accessible to teachers (or digital tutors) can aid in understanding how students are progressing, and suggest who and when needs further assistance. As part of this work, we conducted two studies using computer vision techniques to measure students’ engagement and affective states from their head pose and facial expressions, as they use an online tutoring system, MathSpring.org, designed to aid students’ practice of mathematics problem-solving. We present a Head Pose Tutor, which estimates the real-time head direction of students and responds to potential disengagement, and a Facial Expression-Augmented Teacher Dashboard, that identifies students’ affective states and provides this information to teachers. We collected video data of undergraduate students interacting with MathSpring. Preliminary results on MathSpring videos were encouraging indicating accuracy in detecting head orientation. A usability study was conducted with actual teachers to start to evaluate the possible impact of the proposed Teacher Dashboard software.