Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangement
CoRR(2023)
摘要
Language is compositional; an instruction can express multiple relation
constraints to hold among objects in a scene that a robot is tasked to
rearrange. Our focus in this work is an instructable scene-rearranging
framework that generalizes to longer instructions and to spatial concept
compositions never seen at training time. We propose to represent
language-instructed spatial concepts with energy functions over relative object
arrangements. A language parser maps instructions to corresponding energy
functions and an open-vocabulary visual-language model grounds their arguments
to relevant objects in the scene. We generate goal scene configurations by
gradient descent on the sum of energy functions, one per language predicate in
the instruction. Local vision-based policies then re-locate objects to the
inferred goal locations. We test our model on established instruction-guided
manipulation benchmarks, as well as benchmarks of compositional instructions we
introduce. We show our model can execute highly compositional instructions
zero-shot in simulation and in the real world. It outperforms
language-to-action reactive policies and Large Language Model planners by a
large margin, especially for long instructions that involve compositions of
multiple spatial concepts. Simulation and real-world robot execution videos, as
well as our code and datasets are publicly available on our website:
https://ebmplanner.github.io.
更多查看译文
关键词
compositional scene
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要