Video you only look once: Overall temporal convolutions for action recognition.

•An efficient approach is proposed to capture the overall temporal dynamics from an entire video.•A compact proxy video is generated to preserve the original video temporal dynamics.•3DCNN is employed to learn the overall temporal characteristics from the proxy video.•Extensive experiments demonstrate the effectiveness of the proposed framework.