Self-attention eidetic 3D-LSTM: Video prediction models for traffic flow forecasting

Video prediction is extremely challenging in a traffic flow forecasting problem due to dynamic spatiotemporal dependence. Eidetic 3D convolutional long short-term memory (E3D-LSTM) network, a state-of-the-art video prediction model, proposes a gate-controlled self-attention module called recall gate in the LSTM mechanism to make the present memory state interact with its historical records for long-term relations. Instead of using the gate-controlled self-attention mechanism, we introduce a query-key-value self-attention mechanism into E3D-LSTM for long-term relations from the perspectives of algorithm (internal) and network architecture (external). As for the algorithm (internal) perspective, we replace the original recall gate inside the E3D-LSTM cell with a query-key-value self-attention (SA) module. While for the network architecture (external) perspective, we propose an independent residual query-key-value self-attention (RSA) block outside E3D-LSTM networks in conjunction with the original recall gate. In a traffic flow forecasting problem, we find that both the models from the internal perspective and the external one, named SAE3D-LSTM and RSA-E3D-LSTM, outperform E3D-LSTM and seven other baseline models on three traffic datasets. This validates the effectiveness of the query-key-value self-attention mechanism for long-term relations. Furthermore, we find that SAE3D-LSTM performs better than RSA-E3D-LSTM. This indicates that the query-key-value self-attention mechanism alone can capture long-term relations, dispensing with the gate-controlled self-attention mechanism.