Interactive Multi-Head Self-Attention with Linear Complexity
CoRR(2024)
摘要
We propose an efficient interactive method for multi-head self-attention via
decomposition. For existing methods using multi-head self-attention, the
attention operation of each head is computed independently. However, we show
that the interactions between cross-heads of the attention matrix enhance the
information flow of the attention operation. Considering that the attention
matrix of each head can be seen as a feature of networks, it is beneficial to
establish connectivity between them to capture interactions better. However, a
straightforward approach to capture the interactions between the cross-heads is
computationally prohibitive as the complexity grows substantially with the high
dimension of an attention matrix. In this work, we propose an effective method
to decompose the attention operation into query- and key-less components. This
will result in a more manageable size for the attention matrix, specifically
for the cross-head interactions. Expensive experimental results show that the
proposed cross-head interaction approach performs favorably against existing
efficient attention methods and state-of-the-art backbone models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要