Large Language Model Meets Graph Neural Network in Knowledge Distillation
CoRR(2024)
摘要
Despite recent community revelations about the advancements and potential
applications of Large Language Models (LLMs) in understanding Text-Attributed
Graph (TAG), the deployment of LLMs for production is hindered by its high
computational and storage requirements, as well as long latencies during model
inference. Simultaneously, although traditional Graph Neural Networks (GNNs)
are light weight and adept at learning structural features of graphs, their
ability to grasp the complex semantics in TAG is somewhat constrained for real
applications. To address these limitations, we concentrate on the downstream
task of node classification in TAG and propose a novel graph knowledge
distillation framework, termed Linguistic Graph Knowledge Distillation
(LinguGKD), using LLMs as teacher models and GNNs as student models for
knowledge distillation. It involves TAG-oriented instruction tuning of LLM on
designed tailored prompts, followed by propagating knowledge and aligning the
hierarchically learned node features from the teacher LLM to the student GNN in
latent space, employing a layer-adaptive contrastive learning strategy. Through
extensive experiments on a variety of LLM and GNN models and multiple benchmark
datasets, the proposed LinguGKD significantly boosts the student GNN's
predictive accuracy and convergence rate, without the need of extra data or
model parameters. Compared to teacher LLM, distilled GNN achieves superior
inference speed equipped with much fewer computing and storage demands, when
surpassing the teacher LLM's classification accuracy on some of benchmark
datasets.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要