CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning
arxiv(2024)
摘要
Visual instruction tuning is a key training stage of large multimodal models
(LMMs). Nevertheless, the common practice of indiscriminately mixing
instruction-following data from various tasks may result in suboptimal overall
performance due to different instruction formats and knowledge domains across
tasks. To mitigate this issue, we propose a novel Comprehensive Task Balancing
(CoTBal) algorithm for multi-task visual instruction tuning of LMMs. To our
knowledge, this is the first work that explores multi-task optimization in
visual instruction tuning. Specifically, we consider two key dimensions for
task balancing: (1) Inter-Task Contribution, the phenomenon where learning one
task potentially enhances the performance in other tasks, attributable to the
overlapping knowledge domains, and (2) Intra-Task Difficulty, which refers to
the learning difficulty within a single task. By quantifying these two
dimensions with performance-based metrics, task balancing is thus enabled by
assigning more weights to tasks that offer substantial contributions to others,
receive minimal contributions from others, and also have great intra-task
difficulties. Experiments show that our CoTBal leads to superior overall
performance in multi-task visual instruction tuning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要