PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA
CoRR(2024)
摘要
With the rapid scaling of large language models (LLMs), serving numerous
LoRAs concurrently has become increasingly impractical, leading to unaffordable
costs and necessitating more parameter-efficient finetuning methods. In this
work, we introduce Partially Rotation-enhanced Low-Rank Adaptation (PRoLoRA),
an intra-layer sharing mechanism comprising four essential components:
broadcast reduction, rotation enhancement, partially-sharing refinement, and
rectified initialization strategy. As a superset of LoRA, PRoLoRA pertains its
advantages, and effectively circumvent the drawbacks of peer parameter-sharing
methods with superior model capacity, practical feasibility, and broad
applicability. Empirical experiments demonstrate the remarkably higher
parameter efficiency of PRoLoRA in both specific parameter budget and
performance target scenarios, and its scalability to larger LLMs. Notably, with
one time less trainable parameters, PRoLoRA still outperforms LoRA on multiple
instruction tuning datasets. Subsequently, an ablation study is conducted to
validate the necessity of individual components and highlight the superiority
of PRoLoRA over three potential variants. Hopefully, the conspicuously higher
parameter efficiency can establish PRoLoRA as a resource-friendly alternative
to LoRA.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要