Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code
CoRR(2023)
摘要
We present a method for systematically evaluating the correctness and
robustness of instruction-tuned large language models (LLMs) for code
generation via a new benchmark, Turbulence. Turbulence consists of a large set
of natural language question templates, each of which is a
programming problem, parameterised so that it can be asked in many different
forms. Each question template has an associated test oracle that
judges whether a code solution returned by an LLM is correct. Thus, from a
single question template, it is possible to ask an LLM a
neighbourhood of very similar programming questions, and assess the
correctness of the result returned for each question. This allows gaps in an
LLM's code generation abilities to be identified, including
anomalies where the LLM correctly solves almost all
questions in a neighbourhood but fails for particular parameter instantiations.
We present experiments against five LLMs from OpenAI, Cohere and Meta, each at
two temperature configurations. Our findings show that, across the board,
Turbulence is able to reveal gaps in LLM reasoning ability. This goes beyond
merely highlighting that LLMs sometimes produce wrong code (which is no
surprise): by systematically identifying cases where LLMs are able to solve
some problems in a neighbourhood but do not manage to generalise to solve the
whole neighbourhood, our method is effective at highlighting
robustness issues. We present data and examples that shed light on
the kinds of mistakes that LLMs make when they return incorrect code results.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要