Automatic Pair Construction for Contrastive Post-training
arxiv(2023)
摘要
Alignment serves as an important step to steer large language models (LLMs)
towards human preferences. In this paper, we propose an automatic way to
construct contrastive data for LLM, using preference pairs from multiple models
of varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We compare the
contrastive techniques of SLiC and DPO to SFT baselines and find that DPO
provides a step-function improvement even after continuing SFT saturates. We
also explore a data curriculum learning scheme for contrastive post-training,
which starts by learning from "easier" pairs and transitioning to "harder"
ones, which further improves alignment. Finally, we scale up our experiments to
train with more data and larger models like Orca. Remarkably, our automatic
contrastive post-training further improves the performance of Orca, already a
state-of-the-art instruction learning model tuned with GPT-4 outputs, to
outperform ChatGPT.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要