DevBench: A Comprehensive Benchmark for Software Development
arxiv(2024)
摘要
Recent advancements in large language models (LLMs) have significantly
enhanced their coding capabilities. However, existing benchmarks predominantly
focused on simplified or isolated aspects of programming, such as single-file
code generation or repository issue debugging, falling short of measuring the
full spectrum of challenges raised by real-world programming activities. To
this end, we propose DevBench, a comprehensive benchmark that evaluates LLMs
across various stages of the software development lifecycle, including software
design, environment setup, implementation, acceptance testing, and unit
testing. DevBench features a wide range of programming languages and domains,
high-quality data collection, and carefully designed and verified metrics for
each task. Empirical studies show that current LLMs, including GPT-4-Turbo,
fail to solve the challenges presented within DevBench. Analyses reveal that
models struggle with understanding the complex structures in the repository,
managing the compilation process, and grasping advanced programming concepts.
Our findings offer actionable insights for the future development of LLMs
toward real-world programming applications. Our benchmark is available at
https://github.com/open-compass/DevBench
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要