The First Impression: Understanding the Impact of Multimodal System Responses on User Behavior in Task-oriented Agents

ABSTRACTIn task-oriented dialog agents, the initial user experience sets the tone for future interactions and plays a major role in whether the user returns to the system. But with the diversity of human interaction, these systems often struggle with more complex requests that differ significantly from their training data creating an unpleasant and frustrating user experience. In this work, we formulate an alternative and less resource-intensive approach to improving user experience by implicitly conveying possible user action in system responses. We make use of real-world data to test and support our claims, showing that a significant share of the users made use of such information. Additionally, we extend this to product recommendation as a way to anchor the interaction to well-established flows, we then perform data analysis that shows that close to a third of the users tried benefited from such recommendations. We conclude with an overview of the plans for my Ph.D. thesis and how these findings line up with our goal of producing an ideal agent.