Compressing Deep Models using Multi Tensor Train Decomposition

Deep learning is a very influential research direction in data science and has been applied to various fields in recent years. However, deep neural network usually comes with high computational complexity and large memory storage requirement, thus is difficult to be deployed in mobile devices. The technique of deep neural network compression can reduce the parameter redundancy in depth models by using the low rank and sparsity characteristics of weight matrices or tensors and therefore suitable for applications in mobile systems. In this paper, we employ the Tensor Train decomposition with sparsity constraint for the compression of both convolutional layers and the fully connected layers. Furthermore, the idea of multi Tensor Train decomposition is proposed to improve the performance of the compression model. Specifically, a set of patterns are defined to reshape the original weight matrices or tensors into different shape of high dimensional tensors for compression using Tensor Train decomposition. New network structure is then built based on the decomposition and the number of parameters is highly decreased. Experiments show that the robustness of the compression model increases with the number of patterns employed in the deep model, and the proposed compression approach can achieve a high compression ratio with small performance loss.