Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing
CoRR(2024)
摘要
Large Language Models (LLMs) have demonstrated significant potential in
handling complex reasoning tasks through step-by-step rationale generation.
However, recent studies have raised concerns regarding the hallucination and
flaws in their reasoning process. Substantial efforts are being made to improve
the reliability and faithfulness of the generated rationales. Some approaches
model reasoning as planning, while others focus on annotating for process
supervision. Nevertheless, the planning-based search process often results in
high latency due to the frequent assessment of intermediate reasoning states
and the extensive exploration space. Additionally, supervising the reasoning
process with human annotation is costly and challenging to scale for LLM
training. To address these issues, in this paper, we propose a framework to
learn planning-based reasoning through direct preference optimization (DPO) on
collected trajectories, which are ranked according to synthesized process
rewards. Our results on challenging logical reasoning benchmarks demonstrate
the effectiveness of our learning framework, showing that our 7B model can
surpass the strong counterparts like GPT-3.5-Turbo.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要