Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning
CoRR(2024)
摘要
This paper presents a follow-up study to OpenAI's recent superalignment work
on Weak-to-Strong Generalization (W2SG). Superalignment focuses on ensuring
that high-level AI systems remain consistent with human values and intentions
when dealing with complex, high-risk tasks. The W2SG framework has opened new
possibilities for empirical research in this evolving field. Our study
simulates two phases of superalignment under the W2SG framework: the
development of general superhuman models and the progression towards
superintelligence. In the first phase, based on human supervision, the quality
of weak supervision is enhanced through a combination of scalable oversight and
ensemble learning, reducing the capability gap between weak teachers and strong
students. In the second phase, an automatic alignment evaluator is employed as
the weak supervisor. By recursively updating this auto aligner, the
capabilities of the weak teacher models are synchronously enhanced, achieving
weak-to-strong supervision over stronger student models.We also provide an
initial validation of the proposed approach for the first phase. Using the SciQ
task as example, we explore ensemble learning for weak teacher models through
bagging and boosting. Scalable oversight is explored through two auxiliary
settings: human-AI interaction and AI-AI debate. Additionally, the paper
discusses the impact of improved weak supervision on enhancing weak-to-strong
generalization based on in-context learning. Experiment code and dataset will
be released at https://github.com/ADaM-BJTU/W2SG.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要