FedBRB: An Effective Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learning
CoRR(2024)
摘要
Recently, the success of large models has demonstrated the importance of
scaling up model size. This has spurred interest in exploring collaborative
training of large-scale models from federated learning perspective. Due to
computational constraints, many institutions struggle to train a large-scale
model locally. Thus, training a larger global model using only smaller local
models has become an important scenario (i.e., the small-to-large
scenario). Although recent device-heterogeneity federated learning approaches
have started to explore this area, they face limitations in fully covering the
parameter space of the global model. In this paper, we propose a method called
FedBRB (Block-wise Rolling and weighted
Broadcast) based on the block concept. FedBRB can uses small local
models to train all blocks of the large global model, and broadcasts the
trained parameters to the entire space for faster information interaction.
Experiments demonstrate FedBRB yields substantial performance gains, achieving
state-of-the-art results in this scenario. Moreover, FedBRB using only minimal
local models can even surpass baselines using larger local models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要