VeRA: Vector-based Random Matrix Adaptation
ICLR 2024(2023)
摘要
Low-rank adapation (LoRA) is a popular method that reduces the number of
trainable parameters when finetuning large language models, but still faces
acute storage challenges when scaling to even larger models or deploying
numerous per-user or per-task adapted models. In this work, we present
Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the
number of trainable parameters compared to LoRA, yet maintains the same
performance. It achieves this by using a single pair of low-rank matrices
shared across all layers and learning small scaling vectors instead. We
demonstrate its effectiveness on the GLUE and E2E benchmarks, image
classification tasks, and show its application in instruction-tuning of 7B and
13B language models.
更多查看译文
关键词
Parameter-efficient fine-tuning,Transfer learning,Low-rank,NLP
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要