SP^3: Enhancing Structured Pruning via PCA Projection
arxiv(2023)
摘要
Structured pruning is a widely used technique for reducing the size of
pre-trained language models (PLMs), but current methods often overlook the
potential of compressing the hidden dimension (d) in PLMs, a dimension critical
to model size and efficiency. This paper introduces a novel structured pruning
approach, Structured Pruning with PCA Projection (SP3), targeting the effective
reduction of d by projecting features into a space defined by principal
components before masking. Extensive experiments on benchmarks (GLUE and SQuAD)
show that SP3 can reduce d by 70
over 96
accuracy at the same compression ratio. SP3 has also proven effective with
other models, including OPT and Llama. Our data and code are available at an
anonymous repo.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要