Enhanced Transfer Learning with Efficient Modeling and Adaptive Fusion of Knowledge Via Prompt Tuning

This work presents a novel and parameter-efficient transfer learning framework. The framework consists of two phases: knowledge modeling based on prompt decomposition and knowledge transfer based on attention. Specifically, during the first phase, we decompose the prompt into parameter spaces of different ranks and leverage their characteristics to precisely model both general knowledge and task-specific knowledge separately. During the second phase, we train an attention module to adaptively integrate task-specific knowledge and generate an instance-wise prompt, which is then further fine-tuned. Through these two stages, our approach can accurately and efficiently transfer task knowledge of different granularities and types based on the input sample during inference. Extensive experiments demonstrate that our approach outperforms the multi-task learning baselines and SOTA parameter-efficient transfer learning methods. Furthermore, despite using only 0.13% of the parameters compared to full-parameter fine-tuning, it achieves an absolute improvement ranging from 0.3 to 3.9 across different benchmarks.