Training-time Neuron Alignment through Permutation Subspace for Improving Linear Mode Connectivity and Model Fusion
CoRR(2024)
摘要
In deep learning, stochastic gradient descent often yields functionally
similar yet widely scattered solutions in the weight space even under the same
initialization, causing barriers in the Linear Mode Connectivity (LMC)
landscape. Overcoming these barriers is crucial for understanding deep learning
dynamics and enhancing model-fusion algorithms. Previous studies highlight the
role of permutation symmetry in reducing post-training barriers through network
permutation. However, these post-hoc methods, demanding extra computations, are
less effective for larger, complex models (e.g., ViT, LLM) due to numerous
permutation matrices. Thus, in this paper, we study training-time neuron
alignment. Our hypothesis suggests that training-time permutation subspace can
reduce LMC barriers for free. We find that pruning at initialization supports
this. Beyond pruning, we introduce TNA-PFN, a simple yet lossless algorithm
using a partial gradient mask during training. TNA-PFN is theoretically and
empirically validated for reducing LMC barriers. It excels in wide model fusion
applications, especially in federated learning, two algorithms based on TNA-FPN
that are proposed to show its prospects even under heterogeneous datasets.
Moreover, TNA-PFN can enhance the generalization of model soup for vision
transformers and ColD fusion for pretrained language models.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要