Attention With Sparsity Regularization for Neural Machine Translation and Summarization.

The attention mechanism has become the de facto standard component in neural sequence to sequence tasks, such as machine translation and abstractive summarization. It dynamically determines which parts in the input sentence should be focused on when generating each word in the output sequence. Ideally, only few relevant input words should be attended to at each decoding time step and the attention...