A sub-sampled tensor method for nonconvex optimization

IMA JOURNAL OF NUMERICAL ANALYSIS(2023)

Cited 1|Views9
No score
Abstract
A significant theoretical advantage of high-order optimization methods is their superior convergence guarantees. For instance, third-order regularized methods reach an (epsilon(l), epsilon(2), epsilon(3))third-order critical point in at most 0(max(epsilon(4/3)(1), epsilon(-2)(2), epsilon(-4)(3))) iterations. However, the cost of computing high-order derivatives is prohibitively expensive in real applications, including for instance many real-world machine learning tasks. In order to address this problem, we present a sub-sampled optimization method that uses a thirdorder regularized model to find local minima of smooth and potentially nonconvex objective functions with a finite-sum structure. This algorithm uses sub-sampled derivatives instead of exact quantities and is guaranteed to converge to a third-order critical point. Our analysis relies on a novel tensor concentration inequality for sums of tensors of any order that makes explicit use of the finite-sum structure of the objective function.
More
Translated text
Key words
nonlinear optimization,nonconvex optimization,unconstrained optimization,cubic regularization,trust-region methods
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined