Exploring global diverse attention via pairwise temporal relation for video summarization

•A global diverse attention mechanism is developed to model the long-range temporal dependency of video by using pairwise relations between every two frames regardless of their stride magnitude.•SUM-GDA only needs very limited computational costs by directly handling the pairwise similarity matrix.•SUM-GDA is fully explored in supervised, unsupervised and semi-supervised scenarios. Empirical studies in terms of both quantitative and qualitative views are provided.•The diversity of generated summaries and the influence of optical flow features are both investigated.