Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning
arxiv(2024)
摘要
Through pretraining on a corpus with various sources, Large Language Models
(LLMs) have gained impressive performance. However, the impact of each
component of the pretraining corpus remains opaque. As a result, the
organization of the pretraining corpus is still empirical and may deviate from
the optimal. To address this issue, we systematically analyze the impact of 48
datasets from 5 major categories of pretraining data of LLMs and measure their
impacts on LLMs using benchmarks about nine major categories of model
capabilities. Our analyses provide empirical results about the contribution of
multiple corpora on the performances of LLMs, along with their joint impact
patterns, including complementary, orthogonal, and correlational relationships.
We also identify a set of “high-impact data” such as Books that is
significantly related to a set of model capabilities. These findings provide
insights into the organization of data to support more efficient pretraining of
LLMs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要