Impromptu Accompaniment of Pop Music using Coupled Latent Variable Model with Binary Regularizer

Symbolic music generation has long been an attractive topic in machine intelligence, which aims to automatically learn musical distribution from musical corpora and then to generate samples from the estimated distribution. As one of the most popular kinds, pop music is usually polyphonic and multi-track which makes difficulties in music generation. Furthermore, different from other tasks in machine intelligence such as image processing, the piano roll representation of music is discrete and binary, thus leading to a non-differentiable problem. In other words, most state-of-the-art models such as neural networks cannot be directly applied to achieve piano-roll-based symbolic music generation. To address these two issues, we propose a coupled latent variable model with binary regularizer. On the one hand, the proposed model employs a coupled mechanism to learn a latent variable that simultaneously captures the internal distribution of each track and the joint distribution of multiple tracks. On the other hand, we propose to reformulate the discrete and binary properties into a convex constraint in an elegant way, thus obtaining a differentiable optimization problem and smoothly cooperating with neural networks. To show the effectiveness of our method, we carry out the experiment in impromptu accompaniment generation which is a popular application of music generation, and both the quantitative evaluation and human evaluation demonstrate promising performance of our model compared with some state-of-the-art models.