Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviors
arxiv(2024)
摘要
With the rise of large language models (LLMs), recent works have leveraged
LLMs to improve the performance of click-through rate (CTR) prediction.
However, we argue that a critical obstacle remains in deploying LLMs for
practical use: the efficiency of LLMs when processing long textual user
behaviors. As user sequences grow longer, the current efficiency of LLMs is
inadequate for training on billions of users and items. To break through the
efficiency barrier of LLMs, we propose Behavior Aggregated Hierarchical
Encoding (BAHE) to enhance the efficiency of LLM-based CTR modeling.
Specifically, BAHE proposes a novel hierarchical architecture that decouples
the encoding of user behaviors from inter-behavior interactions. Firstly, to
prevent computational redundancy from repeated encoding of identical user
behaviors, BAHE employs the LLM's pre-trained shallow layers to extract
embeddings of the most granular, atomic user behaviors from extensive user
sequences and stores them in the offline database. Subsequently, the deeper,
trainable layers of the LLM facilitate intricate inter-behavior interactions,
thereby generating comprehensive user embeddings. This separation allows the
learning of high-level user representations to be independent of low-level
behavior encoding, significantly reducing computational complexity. Finally,
these refined user embeddings, in conjunction with correspondingly processed
item embeddings, are incorporated into the CTR model to compute the CTR scores.
Extensive experimental results show that BAHE reduces training time and memory
by five times for CTR models using LLMs, especially with longer user sequences.
BAHE has been deployed in a real-world system, allowing for daily updates of 50
million CTR data on 8 A100 GPUs, making LLMs practical for industrial CTR
prediction.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要