Communication-Efficient Federated Bilevel Optimization with Local and Global Lower Level Problems
NeurIPS 2023(2023)
摘要
Bilevel Optimization has witnessed notable progress recently with new
emerging efficient algorithms. However, its application in the Federated
Learning setting remains relatively underexplored, and the impact of Federated
Learning's inherent challenges on the convergence of bilevel algorithms remain
obscure. In this work, we investigate Federated Bilevel Optimization problems
and propose a communication-efficient algorithm, named FedBiOAcc. The algorithm
leverages an efficient estimation of the hyper-gradient in the distributed
setting and utilizes the momentum-based variance-reduction acceleration.
Remarkably, FedBiOAcc achieves a communication complexity O(ϵ^-1), a
sample complexity O(ϵ^-1.5) and the linear speed up with respect to
the number of clients. We also analyze a special case of the Federated Bilevel
Optimization problems, where lower level problems are locally managed by
clients. We prove that FedBiOAcc-Local, a modified version of FedBiOAcc,
converges at the same rate for this type of problems. Finally, we validate the
proposed algorithms through two real-world tasks: Federated Data-cleaning and
Federated Hyper-representation Learning. Empirical results show superior
performance of our algorithms.
更多查看译文
关键词
global lower bilevel problems,optimization,communication-efficient
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络