A spatially-aware dialogue system for immersive classrooms

Dialogue systems have become a popular research medium as recent advances in task-oriented and open-domain systems combined with deep learning technologies have increased the potential for practical applications across many disciplines. One such vein of applications involves multi-modal dialogue systems deployed in interactive spaces that seek to provide an immersive experience for participants. This project proposes a combination of spatial awareness with a multi-modal, immersive dialogue system as a potential interactive medium to provide an additional layer of immersion. The system employs an array of audio/visual sensors that tracks participants within the interactive space. It responds contextually depending on the application and information domain, for example, by displaying and sonifying conversational agents at accurate spatial locations. The current application of this system involves Mandarin language learning in which the system will act as both a learning medium and conversation augmentation system to provide students with an immersive environment to learn a language and provide real-time feedback during the learning process. This project aims to provide insight into interactive spaces for education and general conversation applications, and demonstrate the capabilities of combining spatial awareness with a multi-modal dialogue system. [Work supported by NSF IIS-1909229, IBM GATOR, and CISL.]