Step-wise Recommendation for Complex Task Support

Digital assistants help people perform simple tasks, including scheduling, home automation, information look up, and question answering. Current assistants offer less support for accomplishing more complex tasks comprising multiple steps. These tasks sometimes require the ability to leverage the capabilities of multiple devices. Ideal smart assistants for such complex scenarios would track task progress and recommend useful and appropriate information at every step of the procedure. To this end, we introduce the novel notion of step-wise recommendation as a means of automatically providing guidance relevant to the current step in a complex task. We employ a common real-life scenario for this purpose: recipe preparation. We demonstrate how a smart assistant can be developed to offer support for complex tasks enabled by multi-modal inputs and outputs through multiple devices (i.e., smart speakers, tablets, or other smart devices available in kitchens). We develop step-wise recommendation models for this scenario and analyze their efficacy for: (1) different prediction tasks (e.g., resources, devices), and (2) different contextual information used to make the prediction (e.g., completed steps, current step, and importantly, future steps). Our recommendation model achieves a prediction accuracy of 83-96%, depending on the prediction task and context used. The findings have implications for the design of intelligent systems to help people accomplish complex tasks.