Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository
arxiv(2024)
摘要
LLMs have demonstrated significant potential in code generation tasks,
achieving promising results at the function or statement level in various
benchmarks. However, the complexities associated with creating code artifacts
like classes, particularly within the context of real-world software
repositories, remain underexplored. Existing research often treats class-level
generation as an isolated task, neglecting the intricate dependencies and
interactions that characterize real-world software development environments. To
address this gap, we introduce RepoClassBench, a benchmark designed to
rigorously evaluate LLMs in generating complex, class-level code within
real-world repositories. RepoClassBench includes natural language to class
generation tasks across Java and Python, from a selection of public
repositories. We ensure that each class in our dataset not only has cross-file
dependencies within the repository but also includes corresponding test cases
to verify its functionality. We find that current models struggle with the
realistic challenges posed by our benchmark, primarily due to their limited
exposure to relevant repository contexts. To address this shortcoming, we
introduce Retrieve-Repotools-Reflect (RRR), a novel approach that equips LLMs
with static analysis tools to iteratively navigate reason about
repository-level context in an agent-based framework. Our experiments
demonstrate that RRR significantly outperforms existing baselines on
RepoClassBench, showcasing its effectiveness across programming languages and
in various settings. Our findings emphasize the need for benchmarks that
incorporate repository-level dependencies to more accurately reflect the
complexities of software development. Our work illustrates the benefits of
leveraging specialized tools to enhance LLMs understanding of repository
context. We plan to make our dataset and evaluation harness public.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要