Human Alignment of Large Language Models through Online Preference Optimisation
arxiv(2024)
摘要
Ensuring alignment of language models' outputs with human preferences is
critical to guarantee a useful, safe, and pleasant user experience. Thus, human
alignment has been extensively studied recently and several methods such as
Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation
(DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper,
our contribution is two-fold. First, we show the equivalence between two recent
alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror
Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD,
that leverages the regularised sampling approach proposed by Nash-MD.
This equivalence may seem surprising at first sight, since IPO is an offline
method whereas Nash-MD is an online method using a preference model. However,
this equivalence can be proven when we consider the online version of IPO, that
is when both generations are sampled by the online policy and annotated by a
trained preference model. Optimising the IPO loss with such a stream of data
becomes then equivalent to finding the Nash equilibrium of the preference model
through self-play. Building on this equivalence, we introduce the IPO-MD
algorithm that generates data with a mixture policy (between the online and
reference policy) similarly as the general Nash-MD algorithm. We compare
online-IPO and IPO-MD to different online versions of existing losses on
preference data such as DPO and SLiC on a summarisation task.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要