Data-Dependent Bounds On Network Gradient Descent
2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton)(2016)
摘要
We study a consensus-based distributed stochastic gradient method for distributed optimization in a setting common for machine learning applications. Nodes in the network hold disjoint data and seek to optimize a common objective which decomposes into a sum of convex functions of individual data points. We show that the rate of convergence for this method involves the spectral properties of two matrices: the standard spectral gap of a weight matrix from the network topology and a new term depending on the spectral norm of the sample covariance matrix of the data. This result shows the benefit of datasets with small spectral norm. Extensions of the method can identify the impact of limited communication, increasing the number of nodes, and scaling with data set size.
更多查看译文
关键词
data-dependent bounds,network gradient descent,consensus-based distributed stochastic gradient method,distributed optimization,machine learning,network nodes,disjoint data,convex functions,individual data points,convergence,matrix spectral properties,weight matrix standard spectral gap,network topology,data covariance matrix
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络