TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
CoRR(2024)
摘要
While tabular classification has traditionally relied on from-scratch
training, a recent breakthrough called prior-data fitted networks (PFNs)
challenges this approach. Similar to large language models, PFNs make use of
pretraining and in-context learning to achieve strong performance on new tasks
in a single forward pass. However, current PFNs have limitations that prohibit
their widespread adoption. Notably, TabPFN achieves very strong performance on
small tabular datasets but is not designed to make predictions for datasets of
size larger than 1000. In this work, we overcome these limitations and
substantially improve the performance of PFNs by developing context
optimization techniques for PFNs. Specifically, we propose TuneTables, a novel
prompt-tuning strategy that compresses large datasets into a smaller learned
context. TuneTables scales TabPFN to be competitive with state-of-the-art
tabular classification methods on larger datasets, while having a substantially
lower inference time than TabPFN. Furthermore, we show that TuneTables can be
used as an interpretability tool and can even be used to mitigate biases by
optimizing a fairness objective.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要