Embodied agents for long-term interaction

An Embodied Conversation Agent (ECA) is a computer interface designed to simulate human face-to-face conversation with its users, through the production of synthesized or prerecorded speech combined with an humanoid embodiment: a representation, either physical or visual, of a body capable of producing some portion of the nonverbal behaviors associated with speech, such as mouth movements, eye movements, head movements, hand gestures, facial expressions, and body posture. A key research problem in the design and implementation of realistic ECAs is generating the range of verbal and nonverbal behavior present in human conversation with appropriate frequency, timing, and quality. ECAs have been used in a variety of applications, motivated by their potential to leverage the affordances of face-to-face conversation to build trust and engagement with users, and their learnability given their use of universally understood communicative cues. Many applications, including education and counseling, are examples of long-term interaction; where an effective agent must have many conversations, over a long period of time, while building rapport with its users. However, prior work on realistic conversational behavior for ECAs has focused heavily on single conversations, isolated from any larger context. To the extent that human conversational behavior is not fixed and unchanging across multiple conversations with the same conversation partner, this approach risks producing ECAs with behavior that becomes increasingly unrealistic in long-term interaction. In this thesis, I present an approach to designing ECAs with realistic verbal and nonverbal behavior in long-term interaction. Based on a longitudinal corpus of health behavior change counseling dialogue, containing multiple conversations between several counselor-client dyads, I construct a series of statistical models demonstrating systematic changes in human conversational behavior across multiple conversations; these changes are predicted both by the interaction history of a dyad, and by the strength or quality of their interpersonal relationship. Based on these findings, I present a model and implementation of verbal and nonverbal behavior generation for ECAs which reproduces some of the observed behavior patterns. Finally, I present a longitudinal randomized controlled evaluation study demonstrating that the resulting model of behavior generation, implemented in an ECA that acts as a virtual health behavior change counselor, produces measurable improvements in user-agent interpersonal bond in long-term interaction.