Orthogonal Neural Network: An Analytical Model for Deep Learning

In the current deep learning model, the computation between each feature and parameter is defined in the real number field. This, together with the nonlinearity of the deep learning model, makes it difficult to analyze the relationship between the values of the computational process and the original features from computation in the real number field. We extend the operational rules of the deep learning model in space and propose the orthogonal neural network (ONN) model, in which the features are set orthogonally to each other in space by "modulating" each input feature of the deep learning model to different orthogonal bases. Because the modulated numerical features are orthogonal to each other, they can be separated from the computations of the ONN model. By "demodulating" the model during and after the calculation, we can obtain a numerical relationship between the results and the original features, which can further provide theoretical and computational support for our analysis of the model. Finally, we compute the weights for each input feature as an interpretable deep learning approach, and describe how the model focuses attention on each feature based on the application of the orthogonal neural network model on two typical models: convolutional neural networks and graph neural networks.