FinanceBench: A New Benchmark for Financial Question Answering.
CoRR(2023)
摘要
FinanceBench is a first-of-its-kind test suite for evaluating the performance
of LLMs on open book financial question answering (QA). It comprises 10,231
questions about publicly traded companies, with corresponding answers and
evidence strings. The questions in FinanceBench are ecologically valid and
cover a diverse set of scenarios. They are intended to be clear-cut and
straightforward to answer to serve as a minimum performance standard. We test
16 state of the art model configurations (including GPT-4-Turbo, Llama2 and
Claude2, with vector stores and long context prompts) on a sample of 150 cases
from FinanceBench, and manually review their answers (n=2,400). The cases are
available open-source. We show that existing LLMs have clear limitations for
financial QA. Notably, GPT-4-Turbo used with a retrieval system incorrectly
answered or refused to answer 81% of questions. While augmentation techniques
such as using longer context window to feed in relevant evidence improve
performance, they are unrealistic for enterprise settings due to increased
latency and cannot support larger financial documents. We find that all models
examined exhibit weaknesses, such as hallucinations, that limit their
suitability for use by enterprises.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要