Solving SDPs for synchronization and MaxCut problems via the Grothendieck inequality

COLT, pp. 1476-1515, 2017.

Cited by: 32|Bibtex|Views40|Links
EI
Keywords:
multiplicative weight update methodnon convexMaxCut SDPgrothendieck type inequalitylow rankMore(19+)
Weibo:
A successful approach to statistical estimation and statistical learning suggests to estimate the object of interest by solving an optimization problem, for instance motivated by maximum likelihood, or empirical risk minimization

Abstract:

A number of statistical estimation problems can be addressed by semidefinite programs (SDP). While SDPs are solvable in polynomial time using interior point methods, in practice generic SDP solvers do not scale well to high-dimensional problems. order to cope with this problem, Burer and Monteiro proposed a non-convex rank-constrained fo...More

Code:

Data:

0
Introduction
  • A successful approach to statistical estimation and statistical learning suggests to estimate the object of interest by solving an optimization problem, for instance motivated by maximum likelihood, or empirical risk minimization.
  • Theorem 2 For any ε-approximate concave point σ ∈ Mk of the rank-k non-convex problem (k-Ncvx-MC-SDP), the authors have f (σ)
  • Theorem 5 For any k ≥ 3, if σ∗ is a local maximizer of the rank-k non-convex SDP problem (13), using σ∗ the authors can find an α∗ × (1 − 1/(k − 1)) ≥ 0.878 × (1 − 1/(k − 1))-approximate solution of the MaxCut problem (11).
Highlights
  • We prove that the error achieved by local maximizers undergoes a phase transition at the same threshold as for information-theoretically optimal methods
  • A successful approach to statistical estimation and statistical learning suggests to estimate the object of interest by solving an optimization problem, for instance motivated by maximum likelihood, or empirical risk minimization
  • We extend our analysis beyond the MaxCut type problem (k-Ncvx-MCSDP) to treat an optimization problem motivated by SO(d) synchronization
  • For the non-convex MaxCut semidefinite programs problem (k-Ncvx-MC-semidefinite programs), we describe the algorithm concretely as follows
  • Theorem 5 For any k ≥ 3, if σ∗ is a local maximizer of the rank-k non-convex semidefinite programs problem (13), using σ∗ we can find an α∗ × (1 − 1/(k − 1)) ≥ 0.878 × (1 − 1/(k − 1))-approximate solution of the MaxCut problem (11)
Results
  • Theorem 6 For any λ > 1, there exists a function k∗(λ) > 0, such that for any k > k∗(λ), with high probability, any local maximizer σ of the rank-k non-convex SDP (k-Ncvx-MC-SDP) problem has non-vanishing correlation with the ground truth parameter.
  • Theorem 9 For an ε-approximate concave point σ ∈ Mo,d,k of the rank-k non-convex Orthogonal-Cut SDP problem (k-Ncvx-OC-SDP), the authors have f (σ)
  • In Figure 2, the authors take A ∼ GOE(1000), and use projected gradient ascent to solve the optimization problem (k-Ncvx-MC-SDP) with a random initialization and fixed step size.
  • Note that Theorem 5 gives a guarantee for the approximation ratio for the cut induced by any local maximizer of the rank-k non-convex SDP.
  • Applying Theorem 2, and noting that the elements of AG are non-negative, the authors for any local maximizer σ∗ of the problem (13), and any X∗ optimal solution of the SDP (12), σ∗, −AGσ∗
  • Let A(λ) = λ/n · uuT + Wn. For any local maximum σ ∈ Crn,k of the rank-k non-convex MaxCut SDP problem, according to Theorem 2, the authors have f (σ)
  • Due to the second order optimality condition, similar to the calculation in Theorem 2, the authors have for any local maximizer σ of the rank-k non-convex SDP problem: 0 ≤ Eg W, (Λ(σ) − A)W =
  • Let σ ∈ Rn×k be a local optimizer of the rank-k non-convex SDP problem.
Conclusion
  • According to (Montanari and Sen, 2016, Theorem 8), the gap between the SDPs with the two different noise matrices is bounded with high probability by a function of the average degree d
  • Let’s consider the case of the rank-k non-convex Orthogonal-Cut SDP problem (k-Ncvx-OC-
  • At each point σ ∈ Mk, Theorem 2 gives the following lower bound on the highest curvature λH,max(σ) = sup u∈Tσ Mk u, Hessf (σ)[u] u, u g(σ) n
Summary
  • A successful approach to statistical estimation and statistical learning suggests to estimate the object of interest by solving an optimization problem, for instance motivated by maximum likelihood, or empirical risk minimization.
  • Theorem 2 For any ε-approximate concave point σ ∈ Mk of the rank-k non-convex problem (k-Ncvx-MC-SDP), the authors have f (σ)
  • Theorem 5 For any k ≥ 3, if σ∗ is a local maximizer of the rank-k non-convex SDP problem (13), using σ∗ the authors can find an α∗ × (1 − 1/(k − 1)) ≥ 0.878 × (1 − 1/(k − 1))-approximate solution of the MaxCut problem (11).
  • Theorem 6 For any λ > 1, there exists a function k∗(λ) > 0, such that for any k > k∗(λ), with high probability, any local maximizer σ of the rank-k non-convex SDP (k-Ncvx-MC-SDP) problem has non-vanishing correlation with the ground truth parameter.
  • Theorem 9 For an ε-approximate concave point σ ∈ Mo,d,k of the rank-k non-convex Orthogonal-Cut SDP problem (k-Ncvx-OC-SDP), the authors have f (σ)
  • In Figure 2, the authors take A ∼ GOE(1000), and use projected gradient ascent to solve the optimization problem (k-Ncvx-MC-SDP) with a random initialization and fixed step size.
  • Note that Theorem 5 gives a guarantee for the approximation ratio for the cut induced by any local maximizer of the rank-k non-convex SDP.
  • Applying Theorem 2, and noting that the elements of AG are non-negative, the authors for any local maximizer σ∗ of the problem (13), and any X∗ optimal solution of the SDP (12), σ∗, −AGσ∗
  • Let A(λ) = λ/n · uuT + Wn. For any local maximum σ ∈ Crn,k of the rank-k non-convex MaxCut SDP problem, according to Theorem 2, the authors have f (σ)
  • Due to the second order optimality condition, similar to the calculation in Theorem 2, the authors have for any local maximizer σ of the rank-k non-convex SDP problem: 0 ≤ Eg W, (Λ(σ) − A)W =
  • Let σ ∈ Rn×k be a local optimizer of the rank-k non-convex SDP problem.
  • According to (Montanari and Sen, 2016, Theorem 8), the gap between the SDPs with the two different noise matrices is bounded with high probability by a function of the average degree d
  • Let’s consider the case of the rank-k non-convex Orthogonal-Cut SDP problem (k-Ncvx-OC-
  • At each point σ ∈ Mk, Theorem 2 gives the following lower bound on the highest curvature λH,max(σ) = sup u∈Tσ Mk u, Hessf (σ)[u] u, u g(σ) n
Funding
  • A.M. was partially supported by the NSF grant CCF-1319979 and DMS-1613091
  • S.M. was supported by Office of Technology Licensing Stanford Graduate Fellowship
Reference
  • Emmanuel Abbe, Afonso S Bandeira, and Georgina Hall. Exact recovery in the stochastic block model. IEEE Transactions on Information Theory, 62(1):471–487, 2016.
    Google ScholarLocate open access versionFindings
  • P-A Absil, Christopher G Baker, and Kyle A Gallivan. Trust-region methods on Riemannian manifolds. Foundations of Computational Mathematics, 7(3):303–330, 2007.
    Google ScholarLocate open access versionFindings
  • Greg W Anderson, Alice Guionnet, and Ofer Zeitouni. An introduction to random matrices, volume 118. Cambridge university press, 2010.
    Google ScholarFindings
  • Mica Arie-Nachimson, Shahar Z Kovalsky, Ira Kemelmacher-Shlizerman, Amit Singer, and Ronen Basri. Global motion estimation from point matches. In 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 2012 Second International Conference on, pages 81–88. IEEE, 2012.
    Google ScholarLocate open access versionFindings
  • Sanjeev Arora and Satyen Kale. A combinatorial, primal-dual approach to semidefinite programs. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pages 227– 236. ACM, 2007.
    Google ScholarLocate open access versionFindings
  • Sanjeev Arora, Elad Hazan, and Satyen Kale. Fast algorithms for approximate semidefinite programming using the multiplicative weights update method. In Foundations of Computer Science, 2005. FOCS 2005. 46th Annual IEEE Symposium on, pages 339–348. IEEE, 2005.
    Google ScholarLocate open access versionFindings
  • Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a metaalgorithm and applications. Theory of Computing, 8(1):121–164, 2012.
    Google ScholarLocate open access versionFindings
  • Jinho Baik, Gerard Ben Arous, Sandrine Peche, et al. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 33(5):1643–1697, 2005.
    Google ScholarLocate open access versionFindings
  • Afonso S Bandeira, Moses Charikar, Amit Singer, and Andy Zhu. Multireference alignment using semidefinite programming. In Proceedings of the 5th conference on Innovations in theoretical computer science, pages 459–470. ACM, 2014.
    Google ScholarLocate open access versionFindings
  • Afonso S Bandeira, Nicolas Boumal, and Vladislav Voroninski. On the low-rank approach for semidefinite programs arising in synchronization and community detection. arXiv preprint arXiv:1602.04426, 2016.
    Findings
  • Alexander I. Barvinok. Problems of distance geometry and convex properties of quadratic maps. Discrete & Computational Geometry, 13(2):189–202, 1995.
    Google ScholarLocate open access versionFindings
  • Nicolas Boumal. A Riemannian low-rank method for optimization over semidefinite matrices with block-diagonal constraints. arXiv:1506.00575, 2015.
    Findings
  • Nicolas Boumal, Vlad Voroninski, and Afonso Bandeira. The non-convex burer-monteiro approach works on smooth semidefinite programs. In Advances in Neural Information Processing Systems, pages 2757–2765, 2016.
    Google ScholarLocate open access versionFindings
  • Samuel Burer and Renato DC Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329–357, 2003.
    Google ScholarLocate open access versionFindings
  • Yash Deshpande, Emmanuel Abbe, and Andrea Montanari. Asymptotic mutual information for the two-groups stochastic block model. arXiv preprint arXiv:1507.08685, 2015.
    Findings
  • Joel Friedman. A proof of alon’s second eigenvalue conjecture. In Proceedings of the thirty-fifth annual ACM symposium on Theory of computing, pages 720–724. ACM, 2003.
    Google ScholarLocate open access versionFindings
  • Dan Garber and Elad Hazan. Approximating semidefinite programs in sublinear time. In Advances in Neural Information Processing Systems, pages 1080–1088, 2011.
    Google ScholarLocate open access versionFindings
  • Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42 (6):1115–1145, 1995.
    Google ScholarLocate open access versionFindings
  • Alexander Grothendieck. Resumede la theorie metrique des produits tensoriels topologiques. Resenhas do Instituto de Matematica e Estatıstica da Universidade de Sao Paulo, 2(4):401–481, 1996.
    Google ScholarLocate open access versionFindings
  • Olivier Guedon and Roman Vershynin. Community detection in sparse networks via grothendiecks inequality. Probability Theory and Related Fields, 165(3-4):1025–1049, 2016.
    Google ScholarLocate open access versionFindings
  • Bruce Hajek, Yihong Wu, and Jiaming Xu. Achieving exact cluster recovery threshold via semidefinite programming. IEEE Transactions on Information Theory, 62(5):2788–2797, 2016.
    Google ScholarLocate open access versionFindings
  • Adel Javanmard, Andrea Montanari, and Federico Ricci-Tersenghi. Phase transitions in semidefinite relaxations. Proceedings of the National Academy of Sciences, 113(16):E2218–E2223, 2016.
    Google ScholarLocate open access versionFindings
  • Iain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. Annals of statistics, pages 295–327, 2001.
    Google ScholarLocate open access versionFindings
  • Subhash Khot and Assaf Naor. Grothendieck-type inequalities in combinatorial optimization. Communications on Pure and Applied Mathematics, 65(7):992–1035, 2012.
    Google ScholarLocate open access versionFindings
  • Subhash Khot, Guy Kindler, Elchanan Mossel, and Ryan O’Donnell. Optimal inapproximability results for max-cut and other 2-variable csps? SIAM Journal on Computing, 37(1):319–357, 2007.
    Google ScholarLocate open access versionFindings
  • Satish Babu Korada and Nicolas Macris. Exact solution of the gauge symmetric p-spin glass model on a complete graph. Journal of Statistical Physics, 136(2):205–230, 2009.
    Google ScholarLocate open access versionFindings
  • J Kuczynski and H Wozniakowski. Estimating the largest eigenvalue by the power and lanczos algorithms with a random start. SIAM journal on matrix analysis and applications, 13(4):1094– 1122, 1992.
    Google ScholarLocate open access versionFindings
  • Laurent Massoulie. Community detection thresholds and the weak ramanujan property. In Proceedings of the 46th Annual ACM Symposium on Theory of Computing, pages 694–703. ACM, 2014.
    Google ScholarLocate open access versionFindings
  • Ankur Moitra, William Perry, and Alexander S Wein. How robust are reconstruction thresholds for community detection? In Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing, pages 828–841. ACM, 2016.
    Google ScholarLocate open access versionFindings
  • Andrea Montanari. A Grothendieck-type inequality for local maxima. arXiv:1603.04064, 2016.
    Findings
  • Andrea Montanari and Subhabrata Sen. Semidefinite programs on sparse random graphs and their application to community detection. In Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing, pages 814–827. ACM, 2016.
    Google ScholarLocate open access versionFindings
  • Elchanan Mossel, Joe Neeman, and Allan Sly. A proof of the block model threshold conjecture. arXiv:1311.4115, 2013.
    Findings
  • Elchanan Mossel, Joe Neeman, and Allan Sly. Reconstruction and estimation in the planted partition model. Probability Theory and Related Fields, 162(3-4):431–461, 2015.
    Google ScholarLocate open access versionFindings
  • Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
    Google ScholarFindings
  • Gabor Pataki. On the rank of extreme matrices in semidefinite programs and the multiplicity of optimal eigenvalues. Mathematics of operations research, 23(2):339–358, 1998.
    Google ScholarLocate open access versionFindings
  • Amit Singer. Angular synchronization by eigenvectors and semidefinite programming. Applied and computational harmonic analysis, 30(1):20–36, 2011.
    Google ScholarLocate open access versionFindings
  • Amit Singer and Yoel Shkolnisky. Three-dimensional structure determination from common lines in cryo-em by eigenvectors and semidefinite programming. SIAM journal on imaging sciences, 4 (2):543–572, 2011.
    Google ScholarFindings
  • David Steurer. Fast sdp algorithms for constraint satisfaction problems. In Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms, pages 684–697. SIAM, 2010.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments