Multimodal correlation deep belief networks for multi-view classification

The Restricted Boltzmann machine (RBM) has been proven to be a powerful tool in many specific applications, such as representational learning, document modeling, and many other learning tasks. However, the extensions of the RBM are rarely used in the field of multi-view learning. In this paper, we present a new RBM model based on canonical correlation analysis, named as the correlation RBM, for multi-view learning. The correlation RBM computes multiple representations by regularizing the marginal likelihood function with the consistency among representations from different views. In addition, the multimodal deep model can obtain a unified representation that fuses multiple representations together. Therefore, we stack the correlation RBM to create the correlation deep belief network (DBN), and then propose the multimodal correlation DBN for learning multi-view data representations. Contrasting with existing multi-view classification methods, such as multi-view Gaussian process with posterior consistency (MvGP) and consensus and complementarity based maximum entropy discrimination (MED-2C), the correlation RBM and the multimodal correlation DBN have achieved satisfactory results on two-class and multi-class classification datasets. Experimental results show that correlation RBM and the multimodal correlation DBN are effective learning algorithms.