Activity Forecasting in Routine Tasks by Combining Local Motion Trajectories and High-Level Temporal Models

Human activity forecasting from videos in routine-based tasks is an open research problem. There are numerous applications in robotics, visual monitoring and skill assessment. Currently, activity forecasting has many challenges because human actions are not fully observable from continuous recording. Additionally, a large number of activities involve fine-grained articulated human motions that are hard to capture using frame-level representations. To overcome these challenges, we propose a method that forecasts human actions by learning the dynamics of local motion patterns. We employed dense trajectories to extract local atomic action patterns and Long Short-Term Memory (LSTM) neural networks for high level action dependencies modeling. The experiments on a public dataset validated the effectiveness of our proposed method in activity forecasting and demonstrated large improvements over the baseline two-stream end-to-end model. We learned that human activity forecasting benefits from learning both the short-range motion patterns and long-term dependencies between actions.