Learning and Recovery in the ReLU Model

Rectified linear units, or ReLUs, have become a preferred activation function for artificial neural networks. In this paper we consider two basic learning problems assuming that the underlying data follow a generative model based on a simple network with ReLU activations. The first problem we study corresponds to learning a generative model in the presence of nonlinearity (modeled by the ReLU functions). Given a set of signal vectors y(i) is an element of R-d, i = 1, 2,..., n, we aim to learn the network parameters, i.e., the d x k matrix A, under the model y(i) = ReLU(Ac-i + b), where b is an element of R-d is a random bias vector. We show that it is possible to recover the column space of A within an error of O(d) (in Frobenius norm) under certain conditions on the distribution of b. The second problem we consider is that of robust recovery of the signal in the presence of outliers. In this setting, we are interested in recovering the latent vector c from its noisy nonlinear images of the form v = ReLU(Ac) + e + w, where e is an element of R-d denotes the outliers with sparsity s and w is an element of R-d denote the dense but small noise. We show that the LASSO algorithm recovers c is an element of R-k within an l(2)-error of O root((k + s) log d)/d) when A is a random Gaussian matrix.