Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures
CoRR(2024)
摘要
Conventional scaling of neural networks typically involves designing a base
network and growing different dimensions like width, depth, etc. of the same by
some predefined scaling factors. We introduce an automated scaling approach
leveraging second-order loss landscape information. Our method is flexible
towards skip connections a mainstay in modern vision transformers. Our
training-aware method jointly scales and trains transformers without additional
training iterations. Motivated by the hypothesis that not all neurons need
uniform depth complexity, our approach embraces depth heterogeneity. Extensive
evaluations on DeiT-S with ImageNet100 show a 2.5
parameter efficiency improvement over conventional scaling. Scaled networks
demonstrate superior performance upon training small scale datasets from
scratch. We introduce the first intact scaling mechanism for vision
transformers, a step towards efficient model scaling.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要