Order-aware Convolutional Pooling for Video Based Action Recognition.

•We propose a novel temporal pooling approach to aggregate the frame-level features of a video, which explores the importance of incorporating the temporal order information.•We propose to treat the temporal evolution of the feature value at each feature dimension as a 1D signal and learn a unique convolutional filter bank for each 1D signal.•The proposed pooling method achieves promising action recognition performance while maintaining a tractable amount of model parameters.