The Role of Gesture in Document Explanation by Embodied Conversational Agents

We describe two empirical studies of how professionals explain documents to lay clients who have varying levels of knowledge about the domain under discussion. We find that hand gestures, and in particular deictic gestures by the professional at various parts of the document play a major role in explanations of documents with clients in face-to-face settings. We describe a preliminary computational model of document explanation by an embodied conversational agent, in which appropriate form and location of hand gestures are used by the agent in explaining a document to a user. Results from a pilot evaluation study indicate that individuals with low levels of domain knowledge prefer receiving explanations from such an agent rather than from a human. Examples are drawn from the healthcare domain, in which research consent forms and hospital discharge instruction forms are used as the documents being explained, and health literacy is used as the measure of client domain knowledge.