Technical Report: The Graph Spectral Token – Enhancing Graph Transformers with Spectral Information
arxiv(2024)
摘要
Graph Transformers have emerged as a powerful alternative to Message-Passing
Graph Neural Networks (MP-GNNs) to address limitations such as over-squashing
of information exchange. However, incorporating graph inductive bias into
transformer architectures remains a significant challenge. In this report, we
propose the Graph Spectral Token, a novel approach to directly encode graph
spectral information, which captures the global structure of the graph, into
the transformer architecture. By parameterizing the auxiliary [CLS] token and
leaving other tokens representing graph nodes, our method seamlessly integrates
spectral information into the learning process. We benchmark the effectiveness
of our approach by enhancing two existing graph transformers, GraphTrans and
SubFormer. The improved GraphTrans, dubbed GraphTrans-Spec, achieves over 10
improvements on large graph benchmark datasets while maintaining efficiency
comparable to MP-GNNs. SubFormer-Spec demonstrates strong performance across
various datasets.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要