LaFFi: Leveraging Hybrid Natural Language Feedback for Fine-tuning Language Models
CoRR(2023)
摘要
Fine-tuning Large Language Models (LLMs) adapts a trained model to specific
downstream tasks, significantly improving task-specific performance. Supervised
Fine-Tuning (SFT) is a common approach, where an LLM is trained to produce
desired answers. However, LLMs trained with SFT sometimes make simple mistakes
and result in hallucinations on reasoning tasks such as question-answering.
Without external feedback, it is difficult for SFT to learn a good mapping
between the question and the desired answer, especially with a small dataset.
This paper introduces an alternative to SFT called Natural Language Feedback
for Finetuning LLMs (LaFFi). LaFFi has LLMs directly predict the feedback they
will receive from an annotator. We find that requiring such reflection can
significantly improve the accuracy in in-domain question-answering tasks,
providing a promising direction for the application of natural language
feedback in the realm of SFT LLMs. Additional ablation studies show that the
portion of human-annotated data in the annotated datasets affects the
fine-tuning performance.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要