Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks
CoRR(2023)
摘要
Our study reveals new theoretical insights into over-smoothing and feature
over-correlation in deep graph neural networks. We show the prevalence of
invariant subspaces, demonstrating a fixed relative behavior that is unaffected
by feature transformations. Our work clarifies recent observations related to
convergence to a constant state and a potential over-separation of node states,
as the amplification of subspaces only depends on the spectrum of the
aggregation function. In linear scenarios, this leads to node representations
being dominated by a low-dimensional subspace with an asymptotic convergence
rate independent of the feature transformations. This causes a rank collapse of
the node representations, resulting in over-smoothing when smooth vectors span
this subspace, and over-correlation even when over-smoothing is avoided. Guided
by our theory, we propose a sum of Kronecker products as a beneficial property
that can provably prevent over-smoothing, over-correlation, and rank collapse.
We empirically extend our insights to the non-linear case, demonstrating the
inability of existing models to capture linearly independent features.
更多查看译文
关键词
graph neural
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要