Robotic Learning From Multimodal Instructions: A Card Game Case Study

This chapter describes the design and evaluation of a learning agent capable of receiving multimodal instructions from a human teaching it how to play a card game. The system design includes a corpus-based grammar, a common representation for both input modalities that is used by the agent's higher-level knowledge, and dialogue components. The initial deployment of the system was as a computer-based agent interacting with human instructors via speech and a touch screen. Our evaluation of this configuration consisted of two experiments involving human users who interacted with the agent using free-flowing speech and card moves executed via the touch screen. The system's recognition of non-verbal input and the pairing of nonverbal input with spoken instructions were virtually flawless. Most problems were out-of-grammar errors. We discuss options for reducing those errors and improving other aspects of the system's operation.