On Learning High Dimensional Structured Single Index Models

Single Index Models (SIMs) are simple yet flexible semi-parametric models for classification and regression, where response variables are modeled as a nonlinear, monotonic function of a linear combination of features. Estimation in this context requires learning both the feature weights and the nonlinear function that relates features to observations. While methods have been described to learn SIMs in the low dimensional regime, a method that can efficiently learn SIMs in high dimensions, and under general structural assumptions, has not been forthcoming. In this paper, we propose computationally efficient algorithms for SIM inference in high dimensions using atomic norm regularization. This general approach to imposing structure in high-dimensional modeling specializes to sparsity, group sparsity, and low-rank assumptions among others. We also provide a scalable, stochastic version of the method. Experiments show that the method we propose enjoys superior predictive performance when compared to generalized linear models such as logistic regression, on several real-world datasets.