Order-Based Pre-training Strategies for Procedural Text Understanding
arxiv(2024)
摘要
In this paper, we propose sequence-based pretraining methods to enhance
procedural understanding in natural language processing. Procedural text,
containing sequential instructions to accomplish a task, is difficult to
understand due to the changing attributes of entities in the context. We focus
on recipes, which are commonly represented as ordered instructions, and use
this order as a supervision signal. Our work is one of the first to compare
several 'order as-supervision' transformer pre-training methods, including
Permutation Classification, Embedding Regression, and Skip-Clip, and shows that
these methods give improved results compared to the baselines and SoTA LLMs on
two downstream Entity-Tracking datasets: NPN-Cooking dataset in recipe domain
and ProPara dataset in open domain. Our proposed methods address the
non-trivial Entity Tracking Task that requires prediction of entity states
across procedure steps, which requires understanding the order of steps. These
methods show an improvement over the best baseline by 1.6
NPN-Cooking and ProPara Datasets respectively across metrics.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要