Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning
arxiv(2022)
摘要
Federated learning aims to train models collaboratively across different
clients without the sharing of data for privacy considerations. However, one
major challenge for this learning paradigm is the data heterogeneity
problem, which refers to the discrepancies between the local data distributions
among various clients. To tackle this problem, we first study how data
heterogeneity affects the representations of the globally aggregated models.
Interestingly, we find that heterogeneous data results in the global model
suffering from severe dimensional collapse, in which representations tend
to reside in a lower-dimensional space instead of the ambient space. Moreover,
we observe a similar phenomenon on models locally trained on each client and
deduce that the dimensional collapse on the global model is inherited from
local models. In addition, we theoretically analyze the gradient flow dynamics
to shed light on how data heterogeneity result in dimensional collapse for
local models. To remedy this problem caused by the data heterogeneity, we
propose FedDecorr, a novel method that can effectively mitigate
dimensional collapse in federated learning. Specifically, FedDecorr
applies a regularization term during local training that encourages different
dimensions of representations to be uncorrelated. FedDecorr, which is
implementation-friendly and computationally-efficient, yields consistent
improvements over baselines on standard benchmark datasets. Code:
https://github.com/bytedance/FedDecorr.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要