Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
CoRR(2024)
摘要
We introduce Generalized Instruction Tuning (called GLAN), a general and
scalable method for instruction tuning of Large Language Models (LLMs). Unlike
prior work that relies on seed examples or existing datasets to construct
instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of
human knowledge and capabilities as input and generates large-scale synthetic
instruction data across all disciplines. Specifically, inspired by the
systematic structure in human education system, we build the taxonomy by
decomposing human knowledge and capabilities to various fields, sub-fields and
ultimately, distinct disciplines semi-automatically, facilitated by LLMs.
Subsequently, we generate a comprehensive list of subjects for every discipline
and proceed to design a syllabus tailored to each subject, again utilizing
LLMs. With the fine-grained key concepts detailed in every class session of the
syllabus, we are able to generate diverse instructions with a broad coverage
across the entire spectrum of human knowledge and skills. Extensive experiments
on large language models (e.g., Mistral) demonstrate that GLAN excels in
multiple dimensions from mathematical reasoning, coding, academic exams,
logical reasoning to general instruction following without using task-specific
training data of these tasks. In addition, GLAN allows for easy customization
and new fields or skills can be added by simply incorporating a new node into
our taxonomy.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要