Maximal Correlation Regression

In this paper, we propose a novel regression analysis approach, called maximal correlation regression, by exploiting the ideas from the Hirschfeld-Gebelein-Renyi (HGR) maximal correlation. We show that in supervised learning problems, the optimal weights in maximal correlation regression can be expressed analytically with the relationships to the HGR maximal correlation functions, which reveals theoretical insights for our approach. In addition, we apply the maximal correlation regression to deep learning, in which efficient training algorithms are proposed for learning the weights in hidden layers. Furthermore, we illustrate that the maximal correlation regression is deeply connected to several existing approaches in information theory and machine learning, including the universal feature selection problem, linear discriminant analysis, and the softmax regression. Finally, experiments on real datasets demonstrate that our approach can obtain performance comparable to the widely used softmax regression based-method.