SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs
CoRR(2024)
摘要
Large language models hold significant potential for integrating various data
types, such as text documents and database records, for advanced analytics.
However, blending text and numerical data presents substantial challenges. LLMs
need to process and cross-reference entities and numbers, handle data
inconsistencies and redundancies, and develop planning capabilities such as
building a working memory for managing complex data queries. In this paper, we
introduce four novel tasks centered around sports data analytics to evaluate
the numerical reasoning and information fusion capabilities of LLMs. These
tasks involve providing LLMs with detailed, play-by-play sports game
descriptions, then challenging them with adversarial scenarios such as new game
rules, longer durations, scrambled narratives, and analyzing key statistics in
game summaries. We conduct extensive experiments on NBA and NFL games to assess
the performance of LLMs on these tasks. Our benchmark, SportsMetrics,
introduces a new mechanism for assessing LLMs' numerical reasoning and fusion
skills.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要