AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
arxiv(2024)
摘要
We present a novel Parameter-Efficient Fine-Tuning (PEFT) method, dubbed as
Adaptive Freezing of Low Rank Adaptation (AFLoRA). Specifically, for each
pre-trained frozen weight tensor, we add a parallel path of trainable low-rank
matrices, namely a down-projection and an up-projection matrix, each of which
is followed by a feature transformation vector. Based on a novel freezing
score, we the incrementally freeze these projection matrices during fine-tuning
to reduce the computation and alleviate over-fitting. Our experimental results
demonstrate that we can achieve state-of-the-art performance with an average
improvement of up to 0.85% as evaluated on GLUE benchmark while yeilding up
to 9.5× fewer average trainable parameters. While compared in terms of
runtime, AFLoRA can yield up to 1.86× improvement as opposed to similar
PEFT alternatives. Besides the practical utility of our approach, we provide
insights on the trainability requirements of LoRA paths at different modules
and the freezing schedule for the different projection matrices. Code will be
released.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要