Multimodal Transformer with Effective History Information Mining for Vision-Based Navigation with Direct Assistance

The Vision-based Navigation with Language-based Assistance (VNLA) is a recently proposed task that requires an agent to navigate according to a high-level language instruction. Due to the lack of step-by-step navigation guidance, the key to VNLA is to conduct goal-oriented exploration, accompanied by querying an advisor when required to obtain direct/indirect assistance to make progress. However, the goal of general Artificial Intelligence (AI) is still to improve the agent's autonomy as much as possible. In this paper, we introduce Multimodal Transformer with Effective History Information Mining (MTHM) to address the unique challenges of this task by incorporating long-term history information into decision-makings. Specifically, MTHM leverages real-time visual observations and encoded language instructions to form memory tokens, which are stored in a variable-length memory bank, and uses these rich history clues for exploration. We demonstrate that encoding all the history information with a Transformer is important to solve the VNLA task, and our approach achieves a new state of the art on the ASKNAV dataset.