Machine Learning Theory Lecture 13

We begin with the most basic setting, in which f is L-Lipschitz with respect to the Euclidean norm. Since f is convex, we have ∂f(x) 6= ∅ for all x ∈ Rn. The algorithm is shown in Algorithm 1. Algorithm 1 Gradient descent for minimizing a convex, 1-Lipschitz function over Rn. 1: procedure GradientDescent(x1 ∈ Rn, T ∈ N) 2: Let η = 1/ √ T 3: for i← 1, ..., T − 1 do 4: xi+1 ← xi − ηgi, where gi = ∇f(xi) if f is differentiable, and otherwise gi is any subgradient in ∂f(xi). 5: return ∑T i=1 xi/T