Learning State Detection with Multimodal Information in Virtual Reality Learning.

The learning state is a crucial factor that significantly impacts students' learning efficiency and overall learning outcomes. It also serves as a vital indicator for evaluating teachers' effectiveness in their instructional practices. To assess the learning state, various aspects of students' behavior, including their expressions, actions, and language, can effectively reveal valuable insights. Traditionally, studies have focused on investigating the learning state in conventional classroom and online education settings. However, limited research has been conducted in the context of virtual reality (VR) immersive learning environments. Considering the unique challenges posed by VR, such as the occlusion of the upper part of the face by VR devices and the distinct characteristics of VR interaction, we propose a novel method for detecting the learning state while wearing VR during the learning process. This method integrates multimodal information, including video captured by a camera, eye movement sensor data from the VR device, and electroencephalography (EEG) signals from an EEG headband. By comprehensively analyzing these inputs, we can determine the learning state effectively. An experimental evaluation was conducted to validate the effectiveness of the proposed multimodal information infusion learning state detection method in the context of VR immersive learning. The results demonstrate that this approach significantly enhances the accuracy of determining the learning state. Ultimately, this advancement has the potential to improve the overall learning quality and facilitate the widespread adoption of VR in educational settings.