BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction
CoRR(2024)
摘要
As a novel and effective fine-tuning paradigm based on large-scale
pre-trained language models (PLMs), prompt-tuning aims to reduce the gap
between downstream tasks and pre-training objectives. While prompt-tuning has
yielded continuous advancements in various tasks, such an approach still
remains a persistent defect: prompt-tuning methods fail to generalize to
specific few-shot patterns. From the perspective of distribution analyses, we
disclose that the intrinsic issues behind the phenomenon are the
over-multitudinous conceptual knowledge contained in PLMs and the abridged
knowledge for target downstream domains, which jointly result in that PLMs
mis-locate the knowledge distributions corresponding to the target domains in
the universal knowledge embedding space. To this end, we intuitively explore to
approximate the unabridged target domains of downstream tasks in a debiased
manner, and then abstract such domains to generate discriminative prompts,
thereby providing the de-ambiguous guidance for PLMs. Guided by such an
intuition, we propose a simple yet effective approach, namely BayesPrompt, to
learn prompts that contain the domain discriminative information against the
interference from domain-irrelevant knowledge. BayesPrompt primitively
leverages known distributions to approximate the debiased factual distributions
of target domains and further uniformly samples certain representative features
from the approximated distributions to generate the ultimate prompts for PLMs.
We provide theoretical insights with the connection to domain adaptation.
Empirically, our method achieves state-of-the-art performance on benchmarks.
更多查看译文
关键词
Prompt,Pre-Trained,Few-shot,Debiased,Domain Abstraction
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要