A deep Koopman operator-based modelling approach for long-term prediction of dynamics with pixel-level measurements

Although previous studies have made some clear leap in learning latent dynamics from high-dimensional representations, the performances in terms of accuracy and inference time of long-term model prediction still need to be improved. In this study, a deep convolutional network based on the Koopman operator (CKNet) is proposed to model non-linear systems with pixel-level measurements for long-term prediction. CKNet adopts an autoencoder network architecture, consisting of an encoder to generate latent states and a linear dynamical model (i.e., the Koopman operator) which evolves in the latent state space spanned by the encoder. The decoder is used to recover images from latent states. According to a multi-step ahead prediction loss function, the system matrices for approximating the Koopman operator are trained synchronously with the autoencoder in a mini-batch manner. In this manner, gradients can be synchronously transmitted to both the system matrices and the autoencoder to help the encoder self-adaptively tune the latent state space in the training process, and the resulting model is time-invariant in the latent space. Therefore, the proposed CKNet has the advantages of less inference time and high accuracy for long-term prediction. Experiments are performed on OpenAI Gym and Mujoco environments, including two and four non-linear forced dynamical systems with continuous action spaces. The experimental results show that CKNet has strong long-term prediction capabilities with sufficient precision.