CodeMind: A Framework to Challenge Large Language Models for Code Reasoning
arxiv(2024)
Abstract
Solely relying on test passing to evaluate Large Language Models (LLMs) for
code synthesis may result in unfair assessment or promoting models with data
leakage. As an alternative, we introduce CodeMind, a framework designed to
gauge the code reasoning abilities of LLMs. CodeMind currently supports three
code reasoning tasks: Independent Execution Reasoning (IER), Dependent
Execution Reasoning (DER), and Specification Reasoning (SR). The first two
evaluate models to predict the execution output of an arbitrary code or code
the model could correctly synthesize. The third one evaluates the extent to
which LLMs implement the specified expected behavior.
Our extensive evaluation of nine LLMs across five benchmarks in two different
programming languages using CodeMind shows that LLMs fairly follow control flow
constructs and, in general, explain how inputs evolve to output, specifically
for simple programs and the ones they can correctly synthesize. However, their
performance drops for code with higher complexity, non-trivial logical and
arithmetic operators, non-primitive types, and API calls. Furthermore, we
observe that, while correlated, specification reasoning (essential for code
synthesis) does not imply execution reasoning (essential for broader
programming tasks such as testing and debugging): ranking LLMs based on test
passing can be different compared to code reasoning.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined