When Transformer Meets Large Graphs: An Expressive and Efficient Two-View Architecture
IEEE Transactions on Knowledge and Data Engineering(2024)
摘要
The successes of applying Transformer to graphs have been witnessed on small graphs (e.g., molecular graphs), yet two barriers prevent its adoption on large graphs (e.g., citation networks). First, despite the benefit of the global receptive field, enormous distant nodes might distract the necessary attention of each target node from its neighborhood. Second, training a Transformer model on large graphs is costly due to the node-to-node attention mechanism's quadratic computational complexity. To break down these barriers, we propose a two-view architecture
Coarformer
, wherein a GNN-based module captures fine-grained local information from the original graph, and a Transformer-based module captures coarse yet long-range information on the coarse graph. We further design a cross-view propagation scheme so that these two views can enhance each other. Our graph isomorphism analysis shows the complementary natures of GNN and Transformer, justifying the motivation and design of
Coarformer
. We conduct extensive experiments on real-world datasets, where
Coarformer
surpasses any single-view method that solely applies a GNN or Transformer. As an ablation,
Coarformer
outperforms straightforward combinations of a GNN model and a Transformer-based model, verifying the effectiveness of our coarse global view and the cross-view propagation scheme. Meanwhile,
Coarformer
consumes the least runtime and GPU memory than those combinations.
更多查看译文
关键词
Graph neural network,representation learning,transformer
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要