Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training(2024)
Abstract
Recent research has explored the creation of questions from code submitted by
students. These Questions about Learners' Code (QLCs) are created through
program analysis, exploring execution paths, and then creating code
comprehension questions from these paths and the broader code structure.
Responding to the questions requires reading and tracing the code, which is
known to support students' learning. At the same time, computing education
researchers have witnessed the emergence of Large Language Models (LLMs) that
have taken the community by storm. Researchers have demonstrated the
applicability of these models especially in the introductory programming
context, outlining their performance in solving introductory programming
problems and their utility in creating new learning resources. In this work, we
explore the capability of the state-of-the-art LLMs (GPT-3.5 and GPT-4) in
answering QLCs that are generated from code that the LLMs have created. Our
results show that although the state-of-the-art LLMs can create programs and
trace program execution when prompted, they easily succumb to similar errors
that have previously been recorded for novice programmers. These results
demonstrate the fallibility of these models and perhaps dampen the expectations
fueled by the recent LLM hype. At the same time, we also highlight future
research possibilities such as using LLMs to mimic students as their behavior
can indeed be similar for some specific tasks.
MoreTranslated text
Key words
QLCs,large language models,artificial intelligence,introductory programming,program comprehension
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined